<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Morsinaldo Medeiros on Medium]]></title>
        <description><![CDATA[Stories by Morsinaldo Medeiros on Medium]]></description>
        <link>https://medium.com/@morsinaldo.medeiros.075?source=rss-d08dfb001c2c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*j-99XzN1_91aA5vo</url>
            <title>Stories by Morsinaldo Medeiros on Medium</title>
            <link>https://medium.com/@morsinaldo.medeiros.075?source=rss-d08dfb001c2c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 04:39:45 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@morsinaldo.medeiros.075/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Uncovering the Eyes of Autonomous Cars: A Journey through Udacity’s Sensor Fusion NanoDegree Course]]></title>
            <link>https://medium.com/@morsinaldo.medeiros.075/uncovering-the-eyes-of-autonomous-cars-a-journey-through-udacitys-sensor-fusion-nanodegree-course-edd66ae7b067?source=rss-d08dfb001c2c------2</link>
            <guid isPermaLink="false">https://medium.com/p/edd66ae7b067</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[sensor-fusion]]></category>
            <category><![CDATA[udacity]]></category>
            <category><![CDATA[autonomous-vehicles]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Morsinaldo Medeiros]]></dc:creator>
            <pubDate>Tue, 23 Jan 2024 17:32:57 GMT</pubDate>
            <atom:updated>2024-01-23T17:32:57.409Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>In the revolutionary realm of autonomous vehicles, the ability to “see” and interpret the surrounding environment has become the backbone of the safety and efficiency of these intelligent machines. In this context, the pressing need to perceive and comprehend the world around an autonomous car has become an incessant quest. After all, in a dynamic ecosystem, the ability to discern obstacles, pedestrians, and other vehicles is crucial for the success of autonomous driving. The figure 1 illustrates Uber’s autonomous car equipped with some of the sensors that will be explored further.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6yMcz_vr_DNccy-71-2rpQ.jpeg" /><figcaption>Figure 1 — Uber&#39;s Autonomous Vehicle. Source: Udacity&#39;s Sensor Fusion NanoDegree Course (2023).</figcaption></figure><p>Each type of sensor has distinct advantages and disadvantages, especially in challenging conditions such as variable lighting and adverse climates. Hence, the need to collect and fuse information from different sensors arises. This enables the machine to create a coherent view of the environment, facilitating real-time decision-making with local processing.</p><p>In this regard, this article aims to discuss the projects from Udacity’s Sensor Fusion NanoDegree course and the implemented solutions. Specifically, this course explores the fusion of data from Lidar, Cameras, and Radars. The code for all project solutions is available on GitHub through the links provided in the respective sections for each project below.</p><h3>Project 1: Lidar Obstacle Detection</h3><p><strong>GitHub Repository: </strong><a href="https://github.com/Morsinaldo/SFND-Lidar-Obstacle-Detection"><strong>SFND-Lidar-Obstacle-Detection</strong></a></p><p>The first project involves developing a system that clusters the detected point clouds by Lidar and performs their grouping. Before delving into the solution, let’s acquaint ourselves with Lidar.</p><h4>What is Lidar?</h4><p>Lidar technology provides detailed data resolution by emitting numerous laser signals. These lasers interact with surrounding objects, reflecting back to the sensor, enabling the calculation of the distance to these objects based on the time it takes for the signals to return. Additionally, insights about the struck object can be gleaned by assessing the intensity of the returned signal. Each laser beam operates within the infrared spectrum and is emitted at various angles, typically covering a full 360-degree range. Although Lidar sensors offer highly precise 3D models of our surroundings, they currently come with a substantial price tag, often exceeding $60,000 for a standard unit.</p><p>Below is a figure featuring some models from the Velodyne brand.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0udje2RLe4uEtL7s" /><figcaption>Figure 2 — Velodyne Lidar models. Source: <a href="https://dewesoft.com/blog/adas-testing-with-velodyne-lidar-data-acquisition">Carsten Frederiksen / Nina Kovač (2021)</a>.</figcaption></figure><p>For example, the HDL-64E model is capable of generating around 100 MB of data per second. Some <strong>advantages</strong> of Lidar include:</p><ol><li><strong>Spatial Precision</strong>: Lidar offers high precision in measuring distances and creating three-dimensional maps of the environment.</li><li><strong>Independence from Lighting Conditions</strong>: It functions well in various lighting conditions, including total darkness, as it utilizes its own laser light.</li></ol><p>Regarding the <strong>disadvantages</strong>, the following stand out:</p><ol><li><strong>High Cost</strong>: Lidars tend to be expensive, which can impact the overall system cost.</li><li><strong>Sensitivity to Weather Conditions</strong>: It can be affected by adverse weather conditions, such as heavy rain or fog.</li></ol><h4>Solution</h4><p>The solution for this project primarily involves implementing a RANSAC (Random Sample Consensus) class that, using Lidar-generated data, identifies points belonging to the same object in a 3D point cloud. To achieve this, a KD-Tree with Euclidean distance is employed to cluster the detected point clouds for each obstacle. Once grouped, the program draws a bounding box around the points. By the end of this project, the execution should yield a result similar to the illustration below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/948/1*-_iNan1pBJbdtZXmdbREWQ.gif" /><figcaption>Figure 3 — Lidar Obstacle Detection result.</figcaption></figure><p>The implemented solutions are divided into three files: euclidean_cluster_kdtree.cpp, kdtree_node.h, and ransac.cpp.</p><h3>Project 2: 2D Feature Tracking</h3><p><strong>GitHub Repository: </strong><a href="https://github.com/Morsinaldo/SFND-2D-feature-tracking"><strong>SFND-2D-feature-tracking</strong></a></p><p>The objective of this project is to create a 2D collision detection system using camera images. To achieve this, it is necessary to detect key points in the image and then execute descriptors that can estimate the distance of these key points. Both detectors and descriptors are already implemented in OpenCV. Therefore, the project involves comparing different combinations to determine which ones yield the best results. Before delving into the solution, let’s familiarize ourselves with cameras in the context of autonomous vehicles.</p><h4>What is a Camera?</h4><p>Cameras are excellent sensors for pattern recognition, traffic sign detection, and object identification based on visual cues. In particular, some autonomous vehicle models implement a surround-view camera system. This technology utilizes multiple cameras strategically positioned around the vehicle to obtain a comprehensive and continuous panoramic view of the surroundings. By seamlessly integrating the feeds from each camera, the surround-view system enhances the vehicle’s perception, minimizing blind spots, and improving overall situational awareness. The figure 4 illustrates a surround-view camera system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nu4sKFJjQfUAsRAC" /><figcaption>Figure 4 — Surround Camera System. Source: <a href="https://electrek.co/2015/05/03/tesla-model-s-test-vehicle-with-multiple-cameras-spotted-near-san-francisco/">Fred Lambert (2015)</a>.</figcaption></figure><p>Some <strong>advantages</strong> of Camera include:</p><ol><li><strong>Detailed Visual Information</strong>: Cameras capture rich visual information, enabling the recognition of objects and patterns.</li><li><strong>Affordable Cost</strong>: Generally, cameras are more cost-effective compared to Lidar sensors.</li></ol><p>Regarding the <strong>disadvantages</strong>, the following stand out:</p><ol><li><strong>Dependence on Lighting</strong>: Cameras can be affected by inadequate lighting conditions, such as total darkness.</li><li><strong>Lower Distance Precision</strong>: Distance precision may be limited compared to sensors like Lidar.</li></ol><h4>Solution</h4><p>The solution to this project was divided into six steps:</p><ol><li><strong>Data Buffer Optimization</strong>: Implemented a dedicated vector to store dataBuffer objects, ensuring the size did not exceed a limit (e.g., 2 elements). A ring buffer was used to optimize memory usage.</li><li><strong>Keypoint Detection</strong>: Implemented the choice of HARRIS, FAST, BRISK, ORB, AKAZE, and SIFT detectors through a string.</li><li><strong>Keypoint Removal</strong>: Removed all keypoints outside a predefined rectangle and used only the keypoints within the rectangle for further processing.</li><li><strong>Keypoint Descriptors</strong>: Implemented the choice of BRIEF, ORB, FREAK, AKAZE, and SIFT descriptors through a string.</li><li><strong>Keypoint Matching</strong>: Implemented FLANN matching, as well as the selection of k-nearest neighbors. Both methods are selectable using the respective strings in the main function.</li><li><strong>Descriptor Distance Ratio</strong>: Used K-Nearest-Neighbor matching to implement the descriptor distance ratio test, analyzing the ratio between the best and second-best match to decide whether to keep an associated pair of keypoints.</li></ol><p>Finally, various combinations of detectors/descriptors were analyzed to determine the best ones. The top three combinations found were:</p><p><strong>FAST (Detector) + BRISK (Descriptor)</strong></p><ul><li>Detection execution time (Average): 11.36345 ms</li><li>Keypoints detected (Average): 1344 points</li><li>Description execution time (Average): 1.5916 ms</li><li>Good Matched Keypoints (Average): 111 points</li></ul><p><strong>FAST (Detector) + BRIEF (Descriptor)</strong></p><ul><li>Detection execution time (Average): 11.62124 ms</li><li>Keypoints detected (Average): 1344 points</li><li>Description execution time (Average): 1.6232 ms</li><li>Good Matched Keypoints (Average): 111 points</li></ul><p><strong>FAST (Detector) + ORB (Descriptor)</strong></p><ul><li>Detection execution time (Average): 12.0863 ms</li><li>Keypoints detected (Average): 1344 points</li><li>Description execution time (Average): 1.64849 ms</li><li>Good Matched Keypoints (Average): 111 points</li></ul><p>The code was implemented in matching2D_Student.cpp file. When executing the program in this step, the result will be similar to that illustrated in the figure 5.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*t-VRipvMtke01Tn-" /><figcaption>Figure 5–2D Feature Tracking result.</figcaption></figure><h3>Project 3: 3D Object Tracking</h3><p><strong>GitHub Repository: </strong><a href="https://github.com/Morsinaldo/SFND-3D-Object-Tracking"><strong>SFND-3D-Object-Tracking</strong></a></p><p>The third project aims to explore the use of deep learning, particularly the YOLO model, for object detection in images. Additionally, it delves into the association of regions in the camera image with Lidar points in a 3D space. This combination of features allows for the calculation of Time-to-Collision (TTC) concerning an obstacle.</p><h4>Solution</h4><p>The solution was divided into four steps:</p><ol><li><strong>Match 3D Points</strong>: Implemented a function called matchBoundingBoxes, which takes previous and current frames as input and provides the ids of corresponding regions of interest as output.</li><li><strong>Compute Lidar-based TTC</strong>: Implemented the computeTTCLidar function, which calculates the TTC in seconds for corresponding 3D objects using only Lidar measurements from corresponding bounding boxes between the current and previous frames.</li><li><strong>Associate Key-point Correspondences with Bounding Boxes</strong>: Implemented the clusterKptMatchesWithROI function, which prepares the TTC calculation based on camera measurements by associating key-point correspondences with enclosing bounding boxes. All correspondences meeting this condition are added to a vector in the respective bounding box.</li><li><strong>Compute Camera-based TTC</strong>: Implemented the computeTTCCamera function, which calculates the TTC in seconds for all corresponding 3D objects using only key-point correspondences from corresponding bounding boxes between the current and previous frames.</li></ol><p>Finally, various combinations of detectors and descriptors were tested, similar to the previous project, to determine which combinations yielded better results. The code was implemented in camFusion_Student.cpp and FinalProject_Camera.cpp files. When executing the program, the result should be similar to the illustration in the figure 6.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SDibuYiozpl3HY5u0hIVbg.png" /><figcaption>Figure 6–3D Object Tracking result.</figcaption></figure><h3>Project 4: Radar Target Generation Detection</h3><p><strong>GitHub Repository: </strong><a href="https://github.com/Morsinaldo/SFND-Radar-Target-Generation-Detection"><strong>SFND-Radar-Target-Generation-Detection</strong></a></p><p>The fourth project employs the 2D CA-CFAR (Cell-Averaging Constant False Alarm Rate) technique to detect target measurements against a background characterized by noise in the context of a Frequency Modulated Continuous Wave (FMCW) radar. Before delving into the actual solution, let’s familiarize ourselves with the Radar sensor and 2D CA-CFAR.</p><h4>What is Radar?</h4><p>Its sparsity and confined range generally characterize radar information, but it provides direct insights into the speed of an object moving in a specific direction. This capability makes radar a highly practical sensor for applications such as cruise control, where understanding the vehicle’s speed in front is crucial. Moreover, radar sensors have become increasingly prevalent and cost-effective in modern vehicles.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*8x7pWk_NSI8WXfDX.jpg" /><figcaption>Figure 7 — Toyota&#39;s Dynamic Radar Cruise Control. Source: <a href="https://www.toyotaofnaperville.com/what-is-dynamic-radar-cruise-control-and-how-do-you-use-it/">Toyota of Naperville (2018)</a></figcaption></figure><p>Some <strong>advantages</strong> of using Radar include:</p><ol><li><strong>Long Range</strong>: Radars have a longer effective range compared to many other sensors.</li><li><strong>Good Performance in Adverse Weather Conditions</strong>: They perform well in adverse weather conditions, such as rain or fog.</li></ol><p>Regarding the <strong>disadvantages</strong>, the following stand out:</p><ol><li><strong>Lower Spatial Precision</strong>: Generally, they offer lower precision in spatial resolution compared to Lidar.</li><li><strong>Difficulty in Identifying Visual Details</strong>: They have difficulty providing detailed visual information for object recognition.</li></ol><h4>What is 2D CA-CFAR?</h4><p>2D CA-CFAR (Cell-Averaging Constant False Alarm Rate) is a method used in radar processing for target detection in a variable background environment. This algorithm operates in two dimensions, evaluating cells or pixels in a two-dimensional matrix, typically representing the radar image. It is designed to dynamically adapt the detection threshold based on background noise characteristics, maintaining a constant rate of false alarms. By averaging the amplitudes of neighboring cells, 2D CA-CFAR adjusts the detection threshold to ensure sensitivity to real targets while maintaining robustness against interferences and variations in the surrounding environment. For more information about this method, you can refer to the following links:</p><ul><li><a href="https://www.mathworks.com/help/phased/ref/2dcfardetector.html">2D CFAR Detector — MathWorks</a></li><li><a href="https://ieeexplore.ieee.org/abstract/document/6930826?casa_token=xCYZk0_JX1YAAAAA:BfI152RWvSJInyntyq0oYhMF1vfXM5FfpYJndBq5_hXPIzsRVjTldXLZ13daQsYsYZ1_VbyPGR4">Adaptive Constant False Alarm Rate (CFAR) Detection in Radar Systems — IEEE Xplore</a></li></ul><h4>Solution</h4><p>The solution for this project was implemented in MATLAB and consists of four steps:</p><ol><li><strong>FMCW Waveform Design</strong>: Based on system requirements, an FMCW waveform was designed with a slope around 2e13. This involved determining the bandwidth (B) and chirp time (Tchirp).</li><li><strong>Simulation Loop</strong>: The target’s movement was simulated, and the mixed signal was calculated for each timestamp. The signal was calculated in a way that, once the Range Fast Fourier Transform (FFT) was implemented, it provided the correct range, i.e., the initial position of the target with an error margin of +/- 10 meters.</li><li><strong>Range FFT (1st FFT)</strong>: The Range FFT was implemented to process the signal from the previous step and plot the result. The outcome of this step can be seen in the figure 8.</li><li><strong>2D CFAR</strong>: In this step, the 2D CFAR was implemented on the output of the 2D FFT operation, i.e., the Range-Doppler Map. This processing is capable of suppressing noise and separating the target signal. The result of this step is illustrated in the figure 9.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/553/1*0WXhwN9vBfWXk8viP2ZWaQ.png" /><figcaption>Figure 8 — Range FFT.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qyoESGxP6kO4wDFffQ3BmQ.png" /><figcaption>Figure 9 — Range-Doppler Map.</figcaption></figure><p>The final outcome, as illustrated in the figures, showed a sharper spike at 100 meters and 37 m/s.</p><h3>Project 5: Unscented Kalman Filter</h3><p><strong>GitHub Repository: </strong><a href="https://github.com/Morsinaldo/SFND-Unscented-Kalman-Filter"><strong>SFND-Unscented-Kalman-Filter</strong></a></p><p>The fifth and final project involves implementing the Unscented Kalman Filter to estimate the state of multiple cars on a highway using noisy Lidar and Radar measurements. The files create a highway with 3 lanes and 3 cars, which are accelerating and changing direction to switch lanes. The project’s goal is to calculate the distance and velocity magnitude from sensor fusions. The estimation error is calculated using the root mean-square error (RMSE), and project approval depends on keeping this error below certain thresholds. Before delving into the actual solution, let’s familiarize ourselves with the Unscented Kalman Filter.</p><h4>What is an Unscented Kalman Filter?</h4><p>The Unscented Kalman Filter (UKF) is a sophisticated extension of the traditional Kalman filter, designed to handle nonlinear and non-Gaussian systems. In the context of autonomous vehicles, the UKF is employed in sensor data fusion, such as Lidar, cameras, and radars, to accurately estimate the state of the vehicle and its surrounding environment. The ability to handle nonlinearities and complex uncertainties makes the UKF a crucial tool for obtaining more robust and reliable estimates in dynamic and challenging environments.</p><p>Unlike the extended Kalman filter, which linearizes the system dynamics, the UKF adopts a nonlinear approach, employing a technique called sigma-point sampling. This allows for a more accurate and effective representation of probability distributions, making it particularly efficient in situations where traditional linearization could introduce significant errors.</p><p>For more information on the theoretical foundation of the UKF, you can refer to the following links:</p><ul><li><a href="https://groups.seas.harvard.edu/courses/cs281/papers/unscented.pdf">The Unscented Kalman Filter</a></li><li><a href="https://towardsdatascience.com/the-unscented-kalman-filter-anything-ekf-can-do-i-can-do-it-better-ce7c773cf88d">The Unscented Kalman Filter: Anything EKF Can Do, I Can Do It Better</a></li><li><a href="https://www.kalmanfilter.net/ukf.html">Kalman Filter and Unscented Kalman Filter</a></li></ul><h4>Solution</h4><p>In this project, the code for the functions in the ukf.cpp and ukf.h files was completed. The results of the UKF code completion can be seen in the animation below, where each of the three traffic cars is tracked using a Constant Turn Rate and Velocity (CTRV) motion model. Despite the varying acceleration and turn rates of the cars, the UKF is still able to track the objects very well. In general, four functions were implemented:</p><ol><li><strong>ProcessMeasurement</strong>: Responsible for processing the distance from both Lidar and Radar.</li><li><strong>Prediction</strong>: Responsible for estimating the object’s location by modifying the state vector, sigma points, and covariance matrix.</li><li><strong>UpdateLidar</strong>: Responsible for updating the probability of the object’s position using Lidar data. It modifies the state vector and covariance matrix.</li><li><strong>UpdateRadar</strong>: Responsible for updating the probability of the object’s position using Radar data. It modifies the state vector and covariance matrix.</li></ol><p>Additionally, an auxiliary function called NormalizeAngle was implemented to normalize the angle between 0 and 2π. When executing the program in this step, the result will be similar to that illustrated in the figure 10.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*Xy-5xr0wpl_BP576rL4CBw.gif" /><figcaption>Figure 10 — Unscented Kalman Filter result.</figcaption></figure><h3><strong>Conclusions</strong></h3><p>The course provides an excellent understanding of the importance of sensor data fusion for effective perception in autonomous vehicles. The strategic combination of Lidar, cameras and radar enables a robust view of the environment, overcoming challenges such as variable lighting conditions and adverse weather. The projects allow the practical application of the knowledge passed on throughout the modules and are considered to contribute significantly to the understanding of the subject.</p><h3>References</h3><ol><li><a href="https://www.mathworks.com/help/phased/ref/2dcfardetector.html">2D CFAR Detector — MathWorks</a></li><li><a href="https://ieeexplore.ieee.org/abstract/document/6930826?casa_token=xCYZk0_JX1YAAAAA:BfI152RWvSJInyntyq0oYhMF1vfXM5FfpYJndBq5_hXPIzsRVjTldXLZ13daQsYsYZ1_VbyPGR4">Adaptive Constant False Alarm Rate (CFAR) Detection in Radar Systems — IEEE Xplore</a></li><li><a href="https://dewesoft.com/blog/adas-testing-with-velodyne-lidar-data-acquisition">Carsten Frederiksen / Nina Kovač. ADAS and Autonomous Vehicle Testing With Velodyne Lidar Data Acquisition, 2021</a>.</li><li><a href="https://electrek.co/2015/05/03/tesla-model-s-test-vehicle-with-multiple-cameras-spotted-near-san-francisco/">Fred Lambert. Tesla Model S test vehicle with multiple cameras spotted near San Francisco, 2015</a>.</li><li><a href="https://www.kalmanfilter.net/ukf.html">Kalman Filter and Unscented Kalman Filter</a></li><li><a href="https://groups.seas.harvard.edu/courses/cs281/papers/unscented.pdf">The Unscented Kalman Filter</a></li><li><a href="https://towardsdatascience.com/the-unscented-kalman-filter-anything-ekf-can-do-i-can-do-it-better-ce7c773cf88d">The Unscented Kalman Filter: Anything EKF Can Do, I Can Do It Better</a></li><li><a href="https://www.toyotaofnaperville.com/what-is-dynamic-radar-cruise-control-and-how-do-you-use-it/">Toyota of Naperville. What is Dynamic Radar Cruise Control and How Do You Use It?, 2018</a>.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=edd66ae7b067" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Notebooks to Scripts]]></title>
            <link>https://towardsdev.com/from-notebooks-to-scripts-28332f3e93d9?source=rss-d08dfb001c2c------2</link>
            <guid isPermaLink="false">https://medium.com/p/28332f3e93d9</guid>
            <category><![CDATA[mlflow]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Morsinaldo Medeiros]]></dc:creator>
            <pubDate>Wed, 15 Feb 2023 11:16:08 GMT</pubDate>
            <atom:updated>2023-02-20T13:24:43.567Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to build and orchestrate an end-to-end ML pipeline using MLFlow and Hydra</h4><h4><strong>Introduction</strong></h4><p>If you work with data science or machine learning, most probably you have started working with notebooks, either Google Colab or Jupyter Notebooks. They are very useful for experimentation, where you run one cell at a time and observe its results. However, although there are tools that allow you to put models into production using notebooks (like Amazon SageMaker, for example), this is not a good practice, and not all tools allow this. In general, if you are looking to go beyond notebooks and put a model into production using Machine Learning Operations best practices (MLOps), you are in the right place! In this article, I’ll show you how you can move from notebooks to python scripts by creating a multi-step automated execution pipeline using the MLFlow tool in conjunction with Hydra.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6eCm1rlNSnLBRPIXb86zSQ.png" /><figcaption>Figure 1 — Project Workflow</figcaption></figure><p>As shown in figure 1, the two tools orchestrate the pipeline from downloading the data to training and testing the model and storing all the artifacts created in the Weights and Biases platform. For this, I created a small project on Github with all the files used and with descriptions of the steps so you can reproduce my results.</p><p>Link to Project: <a href="https://github.com/Morsinaldo/water_bottle_mlflow">https://github.com/Morsinaldo/water_bottle_mlflow</a></p><h4><strong>Scripts and MLFlow</strong></h4><p>Scripts are very useful when you want to automate processes. For example, if you want to download the data that will be used for training your model, you don’t need to run all the cells in a notebook, you can just put all the commands in one script and run it through the command terminal using a command like <em>python fetch_data.py</em> , for example.</p><p>At this point, you may be asking yourself, “Cool, but what if I want to get data from a different location, will I need to change the url directly in the script every time?”. To answer this question, let’s first introduce you to MLFlow. According to the tool’s own <a href="https://mlflow.org/">website</a> [1], MLflow is an open source platform for managing the ML lifecycle, including experimentation, reproducibility, deployment, and a central registration model. MLflow currently offers four components:</p><ul><li><strong>MLFlow Tracking</strong>: Record and query experiments: code, data, config and results.</li><li><strong>MLFlow Projects</strong>: Package data science code in a format to reproduce runs on any platform</li><li><strong>MLFlow Models</strong>: Deploy Machine Learning models in diverse serving environments</li><li><strong>MLFlow Registry</strong>: Store, annotate, discover and manage models in a central repository.</li></ul><p>With this, MLFlow allows us to run scripts with customizable parameters (such as the download URL of the data :D ) in a controlled virtual environment, such as an Anaconda environment, for example.</p><h4><strong>Basic Usage</strong></h4><p>Let’s assume that initially you want to automate the process of fetching data. So, for this, you will need three basic files:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HiZvI9W2qfT6ktBQ7a5UkQ.png" /><figcaption>Figure 2 — Basic files to MLFlow</figcaption></figure><ul><li><strong>run.py</strong>: This file contains a function called processed_args that takes as parameter a dictionary with the parameters passed on the command line and executes the code for that step. The code 1 shows a basic usage of the <em>run.py</em> file.</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528171%252C%2B184%252C%2B195%252C%2B1%2529%26t%3Dseti%26wt%3Dnone%26l%3Dpython%26width%3D680%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D7px%26ph%3D9px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dimport%252520gdown%25250Aimport%252520zipfile%25250Aimport%252520argparse%25250A%25250Adef%252520process_args%2528args%2529%25253A%25250A%252520%252520gdown.download%2528args.url%25252C%252520output%25252C%252520quiet%25253DFalse%2529%25250A%252520%252520%25250A%252520%252520with%252520zipfile.ZipFile%2528%252522..%25252Fdata.zip%252522%25252C%252520%2527r%2527%2529%252520as%252520zip_ref%25253A%25250A%252520%252520%252520%252520%252520%252520%252520%252520zip_ref.extractall%2528%252522..%25252F%252522%2529%25250A%252520%252520%252520%252520%252520%252520%25250A%252520if%252520__name__%252520%25253D%25253D%252520%252522__main__%252522%25253A%25250A%252520%252520%252520%252520parser%252520%25253D%252520argparse.ArgumentParser%2528%2529%25250A%25250A%252520%252520%252520%252520parser.add_argument%2528%252522--url%252522%25252C%252520type%25253Dstr%25252C%252520required%25253DTrue%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520help%25253D%252522url%252520to%252520download%252520the%252520dataset%252522%2529%25250A%25250A%252520%252520%252520%252520ARGS%252520%25253D%252520parser.parse_args%2528%2529%25250A%25250A%252520%252520%252520%252520process_args%2528ARGS%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528171%25252C%2B184%25252C%2B195%25252C%2B1%252529%26t%3Dseti%26wt%3Dnone%26l%3Dpython%26width%3D680%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D7px%26ph%3D9px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dimport%25252520gdown%2525250Aimport%25252520zipfile%2525250Aimport%25252520argparse%2525250A%2525250Adef%25252520process_args%252528args%252529%2525253A%2525250A%25252520%25252520gdown.download%252528args.url%2525252C%25252520output%2525252C%25252520quiet%2525253DFalse%252529%2525250A%25252520%25252520%2525250A%25252520%25252520with%25252520zipfile.ZipFile%252528%25252522..%2525252Fdata.zip%25252522%2525252C%25252520%252527r%252527%252529%25252520as%25252520zip_ref%2525253A%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520zip_ref.extractall%252528%25252522..%2525252F%25252522%252529%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%2525250A%25252520if%25252520__name__%25252520%2525253D%2525253D%25252520%25252522__main__%25252522%2525253A%2525250A%25252520%25252520%25252520%25252520parser%25252520%2525253D%25252520argparse.ArgumentParser%252528%252529%2525250A%2525250A%25252520%25252520%25252520%25252520parser.add_argument%252528%25252522--url%25252522%2525252C%25252520type%2525253Dstr%2525252C%25252520required%2525253DTrue%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520help%2525253D%25252522url%25252520to%25252520download%25252520the%25252520dataset%25252522%252529%2525250A%2525250A%25252520%25252520%25252520%25252520ARGS%25252520%2525253D%25252520parser.parse_args%252528%252529%2525250A%2525250A%25252520%25252520%25252520%25252520process_args%252528ARGS%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/94ca3f21bf50d19e8901de9728d92d57/href">https://medium.com/media/94ca3f21bf50d19e8901de9728d92d57/href</a></iframe><ul><li><strong>conda.yml</strong>: This file contains the libraries (and their respective versions) that will be used to create the virtual environment in which the <em>run.py </em>file will be executed. The code 2 shows a basic usage of the <em>conda.yml</em> file.</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528171%252C%2B184%252C%2B195%252C%2B1%2529%26t%3Dseti%26wt%3Dnone%26l%3Dyaml%26width%3D680%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D7px%26ph%3D9px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dname%25253A%252520fetch_data%25250Achannels%25253A%25250A%252520%252520-%252520conda-forge%25250A%252520%252520-%252520defaults%25250Adependencies%25253A%25250A%252520%252520-%252520pip%25253D22.3.1%25250A%252520%252520-%252520imutils%25253D0.5.4%25250A%252520%252520-%252520pip%25253A%25250A%252520%252520%252520%252520%252520%252520-%252520protobuf%25253D%25253D3.20.1%25250A%252520%252520%252520%252520%252520%252520-%252520gdown%25253D%25253D4.6.0%25250A%252520%252520%252520%252520%252520%252520-%252520zipfile36&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528171%25252C%2B184%25252C%2B195%25252C%2B1%252529%26t%3Dseti%26wt%3Dnone%26l%3Dyaml%26width%3D680%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D7px%26ph%3D9px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dname%2525253A%25252520fetch_data%2525250Achannels%2525253A%2525250A%25252520%25252520-%25252520conda-forge%2525250A%25252520%25252520-%25252520defaults%2525250Adependencies%2525253A%2525250A%25252520%25252520-%25252520pip%2525253D22.3.1%2525250A%25252520%25252520-%25252520imutils%2525253D0.5.4%2525250A%25252520%25252520-%25252520pip%2525253A%2525250A%25252520%25252520%25252520%25252520%25252520%25252520-%25252520protobuf%2525253D%2525253D3.20.1%2525250A%25252520%25252520%25252520%25252520%25252520%25252520-%25252520gdown%2525253D%2525253D4.6.0%2525250A%25252520%25252520%25252520%25252520%25252520%25252520-%25252520zipfile36&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/fc49060a10db38cc2287edc59084bf9e/href">https://medium.com/media/fc49060a10db38cc2287edc59084bf9e/href</a></iframe><ul><li><strong>MLProject</strong>: This file contains the processing parameters that will be passed to the file <em>run.py</em>, as well as the command to run it. For example, the file contains the names of the parameters, their descriptions, their types, whether they are mandatory or not, what their default values are, etc. The code 3 shows a basic usage of the <em>MLProject</em> file.</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528171%252C%2B184%252C%2B195%252C%2B1%2529%26t%3Dseti%26wt%3Dnone%26l%3Dyaml%26width%3D680%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D7px%26ph%3D9px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dname%25253A%252520fetch_data%25250Aconda_env%25253A%252520conda.yml%25250A%25250Aentry_points%25253A%25250A%252520%252520%252520%252520main%25253A%25250A%252520%252520%252520%252520%252520%252520%252520%252520parameters%25253A%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520url%25253A%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520description%25253A%252520url%252520that%252520from%252520the%252520dataset%252520will%252520be%252520downloaded%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520type%25253A%252520str%25250A%25250A%252520%252520%252520%252520%252520%252520%252520%252520command%25253A%252520%25253E-%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520python%252520run.py%252520--url%252520%25257Burl%25257D&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528171%25252C%2B184%25252C%2B195%25252C%2B1%252529%26t%3Dseti%26wt%3Dnone%26l%3Dyaml%26width%3D680%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D7px%26ph%3D9px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dname%2525253A%25252520fetch_data%2525250Aconda_env%2525253A%25252520conda.yml%2525250A%2525250Aentry_points%2525253A%2525250A%25252520%25252520%25252520%25252520main%2525253A%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520parameters%2525253A%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520url%2525253A%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520description%2525253A%25252520url%25252520that%25252520from%25252520the%25252520dataset%25252520will%25252520be%25252520downloaded%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520type%2525253A%25252520str%2525250A%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520command%2525253A%25252520%2525253E-%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520python%25252520run.py%25252520--url%25252520%2525257Burl%2525257D&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/dfb89abf8b7e981fb22e348a16610f71/href">https://medium.com/media/dfb89abf8b7e981fb22e348a16610f71/href</a></iframe><p>To run the files shown above, you can use the following command, assuming you are in the same folder where the files are located.</p><p>mlflow run . -P url=https://drive.google.com/uc?id=ni7cicwixi1g737c</p><p>In my project the MLProject files contain only the entry point main, but you can add others. Take the fictitious MLProject file shown in code 4 as an example.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528171%252C%2B184%252C%2B195%252C%2B1%2529%26t%3Dseti%26wt%3Dnone%26l%3Dyaml%26width%3D680%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D7px%26ph%3D9px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dname%25253A%252520fetch_data%25250Aconda_env%25253A%252520conda.yml%25250A%25250Aentry_points%25253A%25250A%252520%252520%252520%252520main%25253A%25250A%252520%252520%252520%252520%252520%252520%252520%252520parameters%25253A%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520url%25253A%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520description%25253A%252520url%252520that%252520from%252520the%252520dataset%252520will%252520be%252520downloaded%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520type%25253A%252520str%25250A%252509%25250A%252520%252520%252520%252520%252520%252520%252520%252520command%25253A%252520%25253E-%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520python%252520run.py%252520--url%252520%25257Burl%25257D%25250A%252520%252520%252520%252520%25250A%252520%252520%252520%252520secondary%25253A%25250A%252520%252520%252520%252520%252520%252520%252520%252520parameters%25253A%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520url%25253A%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520description%25253A%252520url%252520that%252520from%252520the%252520dataset%252520will%252520be%252520downloaded%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520type%25253A%252520str%25250A%252520%252520%252520%252520%252520%252520%252520%252520command%25253A%252520%25253E-%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252509python%252520run_2.py%252520--url%252520%25257Burl%25257D%25250A&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528171%25252C%2B184%25252C%2B195%25252C%2B1%252529%26t%3Dseti%26wt%3Dnone%26l%3Dyaml%26width%3D680%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D7px%26ph%3D9px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dname%2525253A%25252520fetch_data%2525250Aconda_env%2525253A%25252520conda.yml%2525250A%2525250Aentry_points%2525253A%2525250A%25252520%25252520%25252520%25252520main%2525253A%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520parameters%2525253A%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520url%2525253A%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520description%2525253A%25252520url%25252520that%25252520from%25252520the%25252520dataset%25252520will%25252520be%25252520downloaded%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520type%2525253A%25252520str%2525250A%25252509%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520command%2525253A%25252520%2525253E-%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520python%25252520run.py%25252520--url%25252520%2525257Burl%2525257D%2525250A%25252520%25252520%25252520%25252520%2525250A%25252520%25252520%25252520%25252520secondary%2525253A%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520parameters%2525253A%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520url%2525253A%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520description%2525253A%25252520url%25252520that%25252520from%25252520the%25252520dataset%25252520will%25252520be%25252520downloaded%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520type%2525253A%25252520str%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520command%2525253A%25252520%2525253E-%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252509python%25252520run_2.py%25252520--url%25252520%2525257Burl%2525257D%2525250A&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/3d75a9f86bba1a11aef096442c3760e6/href">https://medium.com/media/3d75a9f86bba1a11aef096442c3760e6/href</a></iframe><p>If you run the command mlflow run ., it will only run the entry point main. But if you want to skip main and run only secondary, you can use the command mlflow run . -e secondary. For more information you can go to the <a href="https://mlflow.org/docs/latest/cli.html">documentation</a> of the tool [2]. At this point, you may ask yourself, “So how can I run multiple steps sequentially?” It is at this point that we will meet the friends of friends: Pleasure, Hydra, friend of MLFlow.</p><h4>Hydra</h4><p>Hydra is a hyperparameter management tool released by Facebook AI Research, which allows users to define hyperparameters using YAML files in a hierarchically structured order. It is also possible to change and execute configuration values directly from the command line and perform a grid search on hyperparameters with a single command.</p><h4><strong>Basic Usage</strong></h4><p>To use Hydra together with MLFlow, you will need a main environment that will allow you to orchestrate the execution of each component of your pipeline. In this environment you will have four base files: <em>main.py</em>, <em>conda.yml</em>, <em>MLProject</em> and <em>conda.yaml</em>. The first three follow the same line that was presented earlier.</p><ul><li><strong>conda.yaml</strong>: This file contains the parameters that will be passed to the pipeline components. These values can be unique or they can contain a list of values, which can then be used in the Grid Search function to search for the best hyperparameters. The code 5 shows a basic usage of the <em>conda.yaml</em> file.</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528171%252C%2B184%252C%2B195%252C%2B1%2529%26t%3Dseti%26wt%3Dnone%26l%3Dyaml%26width%3D680%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D7px%26ph%3D9px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dmain%25253A%25250A%252520%252520project_name%25253A%252520my_project_name%25250A%252520%252520experiment_name%25253A%252520dev%25250A%252520%252520execute_steps%25253A%25250A%252520%252520%252520%252520-%252520component_1%25250A%252520%252520%252520%252520-%252520component_2%25250A%252520%252520random_seed%25253A%25252044%25250Adata%25253A%25250A%252520%252520artifact_name%25253A%252520%252522raw_data.csv%252522%25250A%252520%252520dataset%25253A%252520%252522data%252522%25250A%252520%252520file_url%25253A%252520%252522https%25253A%25252F%25252Fdrive.google.com%25252Fuc%25253Fid%25253D1hb9P1KVMcMBLHhJKKU-_FWX_g7uHb74A%252522%25250A%252520%252520%252523%252520Threshold%252520for%252520Kolomorov-Smirnov%252520test%25250A%252520%252520test_size%25253A%2525200.25%25250A%252520%252520val_size%25253A%2525200.25%25250Amodel%25253A%25250A%252520%252520my_model%25253A%25250A%252520%252520%252520%252520batch_size%25253A%25252050%25250A%252520%252520%252520%252520epochs%25253A%25252020%25250A%252520%252520%252520%252520learning_rate%25253A%2525200.0001%25250A%252520%252520export_artifact%25253A%252520%252522model_export%252522&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528171%25252C%2B184%25252C%2B195%25252C%2B1%252529%26t%3Dseti%26wt%3Dnone%26l%3Dyaml%26width%3D680%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D7px%26ph%3D9px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dmain%2525253A%2525250A%25252520%25252520project_name%2525253A%25252520my_project_name%2525250A%25252520%25252520experiment_name%2525253A%25252520dev%2525250A%25252520%25252520execute_steps%2525253A%2525250A%25252520%25252520%25252520%25252520-%25252520component_1%2525250A%25252520%25252520%25252520%25252520-%25252520component_2%2525250A%25252520%25252520random_seed%2525253A%2525252044%2525250Adata%2525253A%2525250A%25252520%25252520artifact_name%2525253A%25252520%25252522raw_data.csv%25252522%2525250A%25252520%25252520dataset%2525253A%25252520%25252522data%25252522%2525250A%25252520%25252520file_url%2525253A%25252520%25252522https%2525253A%2525252F%2525252Fdrive.google.com%2525252Fuc%2525253Fid%2525253D1hb9P1KVMcMBLHhJKKU-_FWX_g7uHb74A%25252522%2525250A%25252520%25252520%25252523%25252520Threshold%25252520for%25252520Kolomorov-Smirnov%25252520test%2525250A%25252520%25252520test_size%2525253A%252525200.25%2525250A%25252520%25252520val_size%2525253A%252525200.25%2525250Amodel%2525253A%2525250A%25252520%25252520my_model%2525253A%2525250A%25252520%25252520%25252520%25252520batch_size%2525253A%2525252050%2525250A%25252520%25252520%25252520%25252520epochs%2525253A%2525252020%2525250A%25252520%25252520%25252520%25252520learning_rate%2525253A%252525200.0001%2525250A%25252520%25252520export_artifact%2525253A%25252520%25252522model_export%25252522&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/8a3af7dce4d5771abd28bbdbea5111e3/href">https://medium.com/media/8a3af7dce4d5771abd28bbdbea5111e3/href</a></iframe><p>In the <em>main.py</em> file we will need to add the hydra decorator to the function that will perform the selected steps on the command line. Code 6 shows the basic use of this file.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528171%252C%2B184%252C%2B195%252C%2B1%2529%26t%3Dseti%26wt%3Dnone%26l%3Dpython%26width%3D680%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D7px%26ph%3D9px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dimport%252520os%25250Aimport%252520mlflow%25250Aimport%252520hydra%25250Afrom%252520omegaconf%252520import%252520DictConfig%25250A%25250A%252523%252520This%252520automatically%252520reads%252520in%252520the%252520configuration%25250A%252540hydra.main%2528config_name%25253D%2527config%2527%2529%25250Adef%252520process_args%2528config%25253A%252520DictConfig%2529%25253A%25250A%25250A%252520%252520%252520%252520%252523%252520You%252520can%252520get%252520the%252520path%252520at%252520the%252520root%252520of%252520the%252520MLflow%252520project%252520with%252520this%25253A%25250A%252520%252520%252520%252520root_path%252520%25253D%252520hydra.utils.get_original_cwd%2528%2529%25250A%25250A%252520%252520%252520%252520if%252520%252522component_1%252522%252520in%252520steps_to_execute%25253A%25250A%25250A%252520%252520%252520%252520%252520%252520%252520%252520_%252520%25253D%252520mlflow.run%2528%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520os.path.join%2528root_path%25252C%252520%252522fetch_data%252522%2529%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252522main%252522%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520parameters%25253D%25257B%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252522url%252522%25253A%252520config%25255B%252522data%252522%25255D%25255B%252522file_url%252522%25255D%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%25257D%25250A%252520%252520%252520%252520%252520%252520%252520%252520%2529%25250A%25250A%252509%252523%252520Add%252520others%252520components%252520here%25250A%252520%252520%252520%252520%252520%252520%252520%252520%25250Aif%252520__name__%252520%25253D%25253D%252520%252522__main__%252522%25253A%25250A%252520%252520%252520%252520process_args%2528%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528171%25252C%2B184%25252C%2B195%25252C%2B1%252529%26t%3Dseti%26wt%3Dnone%26l%3Dpython%26width%3D680%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D7px%26ph%3D9px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dimport%25252520os%2525250Aimport%25252520mlflow%2525250Aimport%25252520hydra%2525250Afrom%25252520omegaconf%25252520import%25252520DictConfig%2525250A%2525250A%25252523%25252520This%25252520automatically%25252520reads%25252520in%25252520the%25252520configuration%2525250A%25252540hydra.main%252528config_name%2525253D%252527config%252527%252529%2525250Adef%25252520process_args%252528config%2525253A%25252520DictConfig%252529%2525253A%2525250A%2525250A%25252520%25252520%25252520%25252520%25252523%25252520You%25252520can%25252520get%25252520the%25252520path%25252520at%25252520the%25252520root%25252520of%25252520the%25252520MLflow%25252520project%25252520with%25252520this%2525253A%2525250A%25252520%25252520%25252520%25252520root_path%25252520%2525253D%25252520hydra.utils.get_original_cwd%252528%252529%2525250A%2525250A%25252520%25252520%25252520%25252520if%25252520%25252522component_1%25252522%25252520in%25252520steps_to_execute%2525253A%2525250A%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520_%25252520%2525253D%25252520mlflow.run%252528%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520os.path.join%252528root_path%2525252C%25252520%25252522fetch_data%25252522%252529%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252522main%25252522%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520parameters%2525253D%2525257B%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252522url%25252522%2525253A%25252520config%2525255B%25252522data%25252522%2525255D%2525255B%25252522file_url%25252522%2525255D%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%2525257D%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252529%2525250A%2525250A%25252509%25252523%25252520Add%25252520others%25252520components%25252520here%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%2525250Aif%25252520__name__%25252520%2525253D%2525253D%25252520%25252522__main__%25252522%2525253A%2525250A%25252520%25252520%25252520%25252520process_args%252528%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/64b038fb24cb13e6cfb597e655d4a4d0/href">https://medium.com/media/64b038fb24cb13e6cfb597e655d4a4d0/href</a></iframe><p>To run the components from the main environment using Hydra and MLFlow, you will use the following command:</p><p>mlflow run . -P hydra_options=&quot;main.execute_steps=&#39;component_1&#39;&quot;</p><p>If you want to perform two or more steps with just one command, you can just separate the components by a comma, like the following code:</p><p>mlflow run . -P hydra_options=&quot;main.execute_steps=&#39;comp_1,comp_2&#39;&quot;</p><h4>My Project</h4><p>Figure 3 shows the pipeline of my Github project in more detail, and from this you can see that it has 5 main components:</p><ol><li>Fetch Data</li><li>Preprocessing</li><li>Data Segregation</li><li>Train</li><li>Test</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*APWJWFZugG74Ax7seW2ODA.png" /><figcaption>Figure 3 — Pipeline</figcaption></figure><p>Overall, this is about training a model for classifying the water level of a bottle. The data was originally collected from Kaggle, but the scripts used are downloading from my Google Drive account, while the trained model was a Vision Transformer from Pytorch. Figure 4 shows some examples of images found in the dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w6GMxfh7fFmbjDN-rsi-TA.png" /><figcaption>Figure 4 — Images from the dataset</figcaption></figure><p>In the <a href="https://github.com/Morsinaldo/water_bottle_mlflow">Github</a> repository you will find all the instructions on how to run and reproduce all the steps of the project. In addition, you will find the training results and the step-by-step instructions on how to run a Streamlit application and use the trained model on my machine to make image predictions. To perform the prediction, the app downloads the model artifact and encoder stored in my Wandb account, loads the model into memory and performs the prediction with the image you put into the application. So you can either run the pipeline with Mlflow, or just run the app.</p><p>You can check out my video teaching how to run and use the Streamlit application at the following link: <a href="https://www.loom.com/share/857ea77395c24f99a95877bd94ba8bf9">https://www.loom.com/share/857ea77395c24f99a95877bd94ba8bf9</a></p><p>Another question you might be asking yourself is, “Why are you using Weights and Biases when MLFlow also stores the generated artifacts and performs step tracking of the executed steps?” Well, the answer is simply for the convenience of it. MLFlow is a bazooka and offers integration with several tools and frameworks, but I find Weights and Biases’ online storage better. I can switch machines depending on where I am, log into my account, download the artifacts, and continue where I left off. That said, feel free to explore more features of MLFLow and build on the introduction made in this article :D.</p><p>This material is based on lectures by Professor Ivanovitch Silva [3] and the book Practical MLOps [4].</p><h4>References</h4><p>[1] MLFlow official Website. Available at: <a href="https://mlflow.org/">https://mlflow.org</a>/ — Accessed February 12, 2023.<br>[2] MLFlow — Command Line Interface documentation. Available at: <a href="https://mlflow.org/docs/latest/cli.html">https://mlflow.org/docs/latest/cli.html</a> — Accessed February 12, 2023.<br>[3] Ivanovitch Silva Github. Available at: <a href="https://github.com/ivanovitchm/mlops">https://github.com/ivanovitchm/mlops</a> — Accessed February 12, 2023.<br>[4] Practical MLOps. Available at: <a href="https://www.oreilly.com/library/view/practical-mlops/9781098103002/">https://www.oreilly.com/library/view/practical-mlops/9781098103002/</a> — Accessed February 12, 2023.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=28332f3e93d9" width="1" height="1" alt=""><hr><p><a href="https://towardsdev.com/from-notebooks-to-scripts-28332f3e93d9">From Notebooks to Scripts</a> was originally published in <a href="https://towardsdev.com/">Towards Dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learn how to easily improve your neural network in practice]]></title>
            <link>https://blog.devops.dev/learn-how-to-easily-improve-your-neural-network-in-practice-419e5402b4fa?source=rss-d08dfb001c2c------2</link>
            <guid isPermaLink="false">https://medium.com/p/419e5402b4fa</guid>
            <category><![CDATA[wandb]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[weights-and-biases]]></category>
            <category><![CDATA[mlops-solution]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Morsinaldo Medeiros]]></dc:creator>
            <pubDate>Sun, 24 Jul 2022 17:53:22 GMT</pubDate>
            <atom:updated>2022-09-05T16:57:00.271Z</atom:updated>
            <content:encoded><![CDATA[<h3>Tips for improving learning and generalization of a neural network</h3><h4>🚀 Introduction</h4><p>In recent years, the field of Deep Learning has been gaining more and more applications and consequently showing its capability. However, the computation involved behind the model is very heavy, and many times people don’t have a clear idea of what the function of each hyperparameter in a neural network is. So they just go around changing hyperparameters and choose the model with the highest accuracy. Knowing what happens behind them is very important, as a clear understanding of them allows the programmer to achieve greater generalization of the model, as well as achieve better optimization of training time. How about we take a look at how the choice of these hyperparameters can improve your network’s performance?</p><p>You can also check out my video in Portuguese on <a href="https://www.loom.com/share/e86f32090c9e47c69e8266cd98bafd42">loom</a>.</p><h4>📋 Objectives</h4><p>The goal of this work is to first show the difference between the various hyperparameters, as it explains what they are. The notebooks created throughout the development of this work are available on <a href="https://github.com/Morsinaldo/credit_risk_nn">GitHub</a> and the artifacts generated throughout these notebooks are saved on the Weights &amp; Biases (W&amp;B or wandb, for short) platform. The deployment of the model will be done on <a href="https://credit-risk-nn.herokuapp.com/">Heroku</a> using FastAPI so that the community can also test and evaluate. The deploying part can be found in more detail in the <a href="https://medium.com/@alessandro.pereira.700/from-classic-models-to-production-models-8d4ab873ac4d">article</a> <a href="https://medium.com/u/6a5adb50676e">ALESSANDRO NETO</a> and I wrote about deploying using Decision Trees. Here we will focus more on neural network optimization.</p><h4><strong>💼 Dataset</strong></h4><p>The dataset chosen for the development of this paper is that of the Lending Club company, which is a peer-to-peer lending company created in 2006 by Renaud Laplanche based in San Francisco, California. The company offers loans to consumers and small businesses through its website, who can borrow funds from individual investors to help finance their loans. The loans are repaid through an investor’s investment in the borrower’s account, increasing the value of their Lending Club account over time. With this, the dataset has information such as the interest rate of the loans, the amount of money lent, the collateral offered, the annual income of the borrower, the purpose of the loan, whether the borrower has paid on time or not, etc. Altogether it has 52 columns, which will be processed for training.</p><h4><strong>📚 Requirements</strong></h4><ul><li>A Google account to access Google Colaboratory</li><li>A W&amp;B account to save the template</li><li>A GitHub account to save files</li><li>A Heroku account to host the application</li></ul><h4><strong>🧠 Theoretical foundation</strong></h4><p>In this section, we will show you some hyperparameters that you can tweak to improve the learning performance and generalization ability of your network, as well as improve stability. It is important to note that this is just a few tips, which have much deeper knowledge behind each of them. If you are interested, search the Internet for examples, study the theory, and have fun exploring best practices for many types of problems.</p><h4>Network Topology</h4><p>One of the first questions that may come to mind when programming a neural network is: how many layers and how many nodes should my network have? After all, you can’t just endlessly add layers in the belief that the model will get better, because you will end up in the case of overfitting. On the other hand, adding too few layers can cause the model to not learn the problem satisfactorily, thus generating underfitting.</p><p>Importantly, increasing the number of layers has a greater impact on learning than increasing the number of nodes. For example, a network with one layer with ten nodes has a lower learning capacity than a network with two layers with five nodes each. There is no magic formula that will work for all cases, but you can find in the literature some values by which you can start to evaluate your model depending on your problem type. With our example it is not very complex, we will use only one hidden layer and vary the number of neurons. As a base, we will start with 100 neurons and change this value throughout the article.</p><h4>Loss function</h4><p>Also known as objective function, it turns the process of learning the model into an optimization problem, and its choice is also very important when improving and optimizing the model. Typically, a neural network model is trained using the stochastic gradient descent algorithm and the weights are updated using the backpropagation of error algorithm.</p><p>This algorithm seeks to update the weights so that the next evaluation reduces the error, meaning that the algorithm is navigating the error gradient. Nice, but we have not yet answered the question: which algorithm do I choose for my network? In the literature, you can find some tips for each problem category. In the book Deep Learning for Natural Language Processing, Jason Brownlee indicates the following combinations:</p><ol><li>Regression problems<br>1.1 <strong>Output layer</strong>: a node with a linear activation function<br>1.2 <strong>Cost function</strong>: Mean Square Error (MSE)</li><li>Binary Classification problem<br>2.1 <strong>Output layer</strong>: a node with sigmoid activation function<br>2.2 <strong>Cost function</strong>: cross-entropy, also known as logarithmic cost</li><li>Multi-class classification problem<br>3.1 <strong>Output layer</strong>: a node for each class with <em>softmax</em> activation function<br>3.2 <strong>Cost function</strong>: cross-entropy, also known as logarithmic cost</li></ol><p>Since our problem is binary classification, we will use the output layer with only one node and the sigmoid activation function.</p><h4>Batch size</h4><p>Another hyperparameter that can be set when training the model is the batch size. Basically, it tells how many examples the algorithm should take to compute the error gradient and update the weights. Using fewer examples results in a less accurate estimate of the error gradient, which consequently results in noisier updates to the model weights. However, these noisy updates can result in faster learning and sometimes a more robust model. Here, we will perform tests with batch size equal to 16, 32 and 64.</p><h4>Learning rate</h4><p>This hyperparameter sets how fast or slow your model will train and learn about the problem. In other words, it controls the amount of distributed error with which the model’s weights are updated each time, such as at the end of each batch of training examples. In general,</p><ul><li>A high learning rate causes the model to learn faster, at the cost of arriving at a suboptimal final set of weights.</li><li>A lower learning rate may allow the model to learn a more optimal or even globally optimal set of weights, but it may take much longer to train.</li></ul><p>Taking the extreme cases, a very large learning rate will result in very large weight updates, and the model performance will oscillate over training epochs. The oscillating performance is caused by weights that diverge (are divergent). At the other extreme, a very small learning rate may never converge, or may get stuck in a suboptimal solution.</p><h4>Epochs</h4><p>The number of epochs refers to the number of complete steps that the model will go through the dataset before it finishes training. Its choice is also very important so that the model does not become underfitted by training too few epochs, nor overfitted by training too many epochs. Also, training for many epochs raises the computational cost, thus taking more time to train. In our example we will train the model for 50, 100 and 150 epochs.</p><h4>Data normalization</h4><p>It is quite common for training datasets to have different units (km, feet, kilometers per hour, etc.), so you must standardize this data in the preprocessing stage. This is another very important part of the neural net training process, as it can make the learning process slow and unstable. Very large numeric variables will generate very large weights, while very scattered output variables can generate gradient bursts due to the high error generated, thus causing drastic changes in the weights. Thus, one should normalize the output variables when they are too spread out. Briefly:</p><ul><li>Standardize the input data</li><li>Normalize the output data</li></ul><p>To perform normalization, you can use the code below:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528252%252C252%252C252%252C1%2529%26t%3Dvscode%26wt%3Dnone%26l%3Dpython%26width%3D680%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D3px%26ph%3D4px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dfrom%252520sklearn.preprocessing%252520import%252520Normalizer%25250A%252523%252520created%252520scaler%25250Anormalizer%252520%25253D%252520Normalizer%2528%2529%25250A%252520%25250A%252523%252520fit%252520normalizer%252520on%252520training%252520dataset%25250Anormalizer.fit%2528train_y%2529%25250A%252520%25250A%252523%252520transform%252520training%252520dataset%25250Atrain_y%252520%25253D%252520normalizer.transform%2528train_y%2529%25250A%252520%25250A%252523%252520transform%252520test%252520dataset%25250Atest_y%252520%25253D%252520normalizer.transform%2528test_y%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528252%25252C252%25252C252%25252C1%252529%26t%3Dvscode%26wt%3Dnone%26l%3Dpython%26width%3D680%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D3px%26ph%3D4px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dfrom%25252520sklearn.preprocessing%25252520import%25252520Normalizer%2525250A%25252523%25252520created%25252520scaler%2525250Anormalizer%25252520%2525253D%25252520Normalizer%252528%252529%2525250A%25252520%2525250A%25252523%25252520fit%25252520normalizer%25252520on%25252520training%25252520dataset%2525250Anormalizer.fit%252528train_y%252529%2525250A%25252520%2525250A%25252523%25252520transform%25252520training%25252520dataset%2525250Atrain_y%25252520%2525253D%25252520normalizer.transform%252528train_y%252529%2525250A%25252520%2525250A%25252523%25252520transform%25252520test%25252520dataset%2525250Atest_y%25252520%2525253D%25252520normalizer.transform%252528test_y%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/d786d5dac376b7ad19160455d2937af9/href">https://medium.com/media/d786d5dac376b7ad19160455d2937af9/href</a></iframe><p>For standardizing the data, two very common ones are StandardScaler and MinMaxScaler. There are others that may even better suit your problem. To use them, you can follow the following code:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528252%252C252%252C252%252C1%2529%26t%3Dvscode%26wt%3Dnone%26l%3Dpython%26width%3D680%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D3px%26ph%3D4px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dfrom%252520sklearn.preprocessing%252520import%252520MinMaxScaler%25250A%252523%252520created%252520scaler%25250Ascaler%252520%25253D%252520MinMaxScaler%2528feature_range%25253D%2528-1%25252C1%2529%2529%25250A%252520%25250A%252523%252520fit%252520scaler%252520on%252520training%252520dataset%25250Ascaler.fit%2528train_y%2529%25250A%252520%25250A%252523%252520transform%252520training%252520dataset%25250Atrain_y%252520%25253D%252520scaler.transform%2528train_y%2529%25250A%252520%25250A%252523%252520transform%252520test%252520dataset%25250Atest_y%252520%25253D%252520scaler.transform%2528test_y%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528252%25252C252%25252C252%25252C1%252529%26t%3Dvscode%26wt%3Dnone%26l%3Dpython%26width%3D680%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D3px%26ph%3D4px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dfrom%25252520sklearn.preprocessing%25252520import%25252520MinMaxScaler%2525250A%25252523%25252520created%25252520scaler%2525250Ascaler%25252520%2525253D%25252520MinMaxScaler%252528feature_range%2525253D%252528-1%2525252C1%252529%252529%2525250A%25252520%2525250A%25252523%25252520fit%25252520scaler%25252520on%25252520training%25252520dataset%2525250Ascaler.fit%252528train_y%252529%2525250A%25252520%2525250A%25252523%25252520transform%25252520training%25252520dataset%2525250Atrain_y%25252520%2525253D%25252520scaler.transform%252528train_y%252529%2525250A%25252520%2525250A%25252523%25252520transform%25252520test%25252520dataset%2525250Atest_y%25252520%2525253D%25252520scaler.transform%252528test_y%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/5509408a84081aa43d52144f87eef330/href">https://medium.com/media/5509408a84081aa43d52144f87eef330/href</a></iframe><p>The <em>feature_range</em> parameter refers to the range you want to standardize. To use StandardScaler, simply import it and change the instance of the object. Also, you can use standardization and normalization simultaneously, as shown in the following code.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f3ce4b9eadc64cab917f81ba04b872c3/href">https://medium.com/media/f3ce4b9eadc64cab917f81ba04b872c3/href</a></iframe><p>With these techniques, you can improve the stability of your model’s learning process and improve the performance of your network.</p><h4>Activation Function — Vanishing gradient</h4><p>In a neural network, the activation function is responsible for transforming the summed weighted input of the node into the node activation or output for that input. If its choice is not appropriate for the problem, vanishing gradient can be generated. This problem occurs when the amount of error decreases dramatically with each additional layer through which it is propagated, given the derivative of the chosen activation function. This prevents multilayer neural networks from learning the problem effectively.</p><p>According to Brownlee (2019), one solution to this problem is to use the rectified linear activation function (ReLU), which is a piecewise linear function that will output the input directly if it is positive, otherwise it will output zero. It has become the default activation function for many categories of neural networks because a model that uses it is easier to train and generally gets better performance.</p><h4>Exploding gradient</h4><p>Another problem that can prevent the network from training effectively is exploding gradient, which occurs due to large updates in the weights. In general, some causes of gradient bursting are:</p><ul><li>Poor choice of learning rate that results in large weight upgrades.</li><li>Poor choice of data preparation, allowing large differences in the target variable.</li><li>Poor choice of loss function, allowing large error values to be calculated.</li></ul><p>A common solution to exploding gradients is to change the derivative of the error before propagating it backwards through the network and use it to update the weights. By rescaling the error derivative, the updates to the weights will also be rescaled, dramatically decreasing the likelihood of bursting or overflowing. There are two main methods for updating the error derivative:</p><ul><li><strong>Gradient scaling</strong>: involves normalizing the error gradient vector so that the vector norm (magnitude) equals a defined value, such as 1.0.</li><li><strong>Gradient Clipping</strong>: Involves forcing the gradient values (in terms of element) to a specific minimum or maximum value if the gradient exceeds an expected range.</li></ul><p>Together, these methods are often simply called gradient clipping. In gradient scaling, we could specify a norm of 1.0, meaning that if the vector norm for a gradient exceeds 1.0, the vector values will be resized so that the vector norm equals 1.0. This can be used in Keras by specifying the <em>clipnorm</em> argument in the optimizer.</p><pre>#configure sgd with gradient norm clipping<br>optimizer = SGD(lr=0.01, momentum=0.9, clipnorm=1.0)</pre><p>In the gradient clipping we can specify a norm of 0.5, which means that if a gradient value is less than -0.5 it will be set to -0.5 and if it is greater than 0.5 it will be set to 0.5. This can be used in Keras by specifying the <em>clipvalue</em> argument in the optimizer. The following code shows the configuration for both cases.</p><pre># configure sgd with gradient value clipping<br>optimizer = SGD(lr=0.01, momentum=0.9, clipvalue=0.5)</pre><h4><strong>Fix overfiting with regularization</strong></h4><p>To solve the underfiting problem, we simply add more complexity to our network to improve its learning ability. To solve overfiting, we need some more complex techniques. Some simpler approaches are:</p><ol><li>Train the model with more examples</li><li>Change network complexity<br>2.1 Change the complexity through the structure (number of weights)<br>2.2 Change the complexity through parameters (weight values)</li></ol><p>However, these techniques still do not involve regularization, but are basically a modification of the algorithm to improve the generalization error and not the training error. Here are some techniques that use regularization</p><h4><strong>Penalize Large weights with weight regularization</strong></h4><p>Perhaps the simplest regularization method is to add a penalty to the cost function proportional to the size of the weights. This will encourage the model to map inputs to outputs in the training data set so that the weights in the model are kept small. The weights of the neural network are real values that can be positive or negative, so simply adding the weights is not enough. There are two main approaches used to calculate the size of the weights:</p><ol><li>Calculate the sum of the absolute values of the weights, called the L1 norm.</li><li>Calculate the sum of the square values of the weights, called the L2 norm.</li></ol><p>L1 encourages weights to 0.0 if possible, resulting in sparser weights (weights with more values of 0.0). L2 punishes larger weights more severely, thus resulting in less sparse weights. In the code you can use the <em>kernel_regularizer</em> parameter to adjust this penalty.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d933011d5d63af820d87fc624ec6becd/href">https://medium.com/media/d933011d5d63af820d87fc624ec6becd/href</a></iframe><p>You can also combine the two penalties and place a penalty for norm, as shown in the following code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/435e39e9a9264112423098f137446050/href">https://medium.com/media/435e39e9a9264112423098f137446050/href</a></iframe><h4><strong>Force small weights with weight constraint</strong></h4><p>A weight constraint is an update to the network that checks the size of the weights (e.g., their norm vector) and if the size exceeds a predefined threshold, the weights are resized so that their size is below the threshold or between a range. Unlike the previous technique, which only encourages the weights to be small, this one ensures that they are.</p><p>A constraint is applied to every node within a layer. All nodes in the layer use the same constraint, and often multiple hidden layers in the same network will use the same constraint. In addition, the constraint can be applied after each update of the weights, for example, at the end of each minibatch. Some constraints that the Keras API supports are listed below:</p><ul><li><strong>Maximum norm</strong> (<em>max_norm</em>), to force weights to have a magnitude at or below a given limit.</li><li><strong>Non-negative norm</strong> (<em>non_neg</em>), to force weights to have a positive magnitude.</li><li><strong>Unit norm</strong> (<em>unit_norm</em>), to force weights to have a magnitude of 1.0.</li><li><strong>Min-Max norm</strong> (<em>min_max_norm</em>), to force weights to have a magnitude between a range.</li></ul><p>In the code, you can set this restriction through the <em>kernel_constraint</em> parameter. Below you can see some code examples.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/698872c7523208e39bfff4b699f55c6a/href">https://medium.com/media/698872c7523208e39bfff4b699f55c6a/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/aba1715e0bc955331369593ee95feb10/href">https://medium.com/media/aba1715e0bc955331369593ee95feb10/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7d03ea5c8464e52313b56e8fd5f8cbd4/href">https://medium.com/media/7d03ea5c8464e52313b56e8fd5f8cbd4/href</a></iframe><h4><strong>Decouple layers with dropout</strong></h4><p>Dropout is a regularization method that approximates training a large number of neural nets with different architectures in parallel. In other words, some of the nodes are randomly dropped during the training process, causing each update of a layer to be performed with a different view of the configured layer. This makes the training process noisy, forcing the nodes within a layer to probabilistically assume more or less responsibility for the inputs and thus making the model more robust.</p><p>This method specifies the probability of retaining the output of each node and can be implemented in any or all layers of the network, but never in the output layer, since that is exactly what we are trying to predict. A common value is a probability of 0.5 to retain the output of each node in a hidden layer, and a value close to 1.0, such as 0.8, to retain the inputs of the visible layer. The code below shows a network implementing dropout after the hidden layer and before the output layer.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/445cece47530104ec85f1bc44982031e/href">https://medium.com/media/445cece47530104ec85f1bc44982031e/href</a></iframe><h4><strong>Promote robustness with noise</strong></h4><p>One approach to improve the generalization error and improve the structure of the mapping problem is to add random noise. Adding random noise increases the number of samples and makes the model less able to remember the training samples, since they are changing all the time. This results in lower weights and a more robust network with lower generalization error.</p><p>Noise can be added to training regardless of the problem being addressed. It is appropriate to try adding noise to classification and regression problems and the type of noise can be specialized to the types of data used as model input, for example, two-dimensional noise in the case of images and signal noise in the case of audio data. Also, it is important to note that it is only added at the training stage and never at the testing stage.</p><p>The following codes show the implementation of a very common noise, the Gaussian noise, which has a mean of 0 and a standard deviation of 1 (these parameters are configurable). Furthermore, this is an example where the input is two-dimensional.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d797973c21c5b2fb15b7c060495a6ce7/href">https://medium.com/media/d797973c21c5b2fb15b7c060495a6ce7/href</a></iframe><h4><strong>Halt training at the right time with early stopping</strong></h4><p>One of the biggest challenges in training neural networks is time, because training for too long can lead to overfitting, while training for too short a time can lead to underfitting. By monitoring the model’s performance in the validation base, it is possible to stop training when the generalization error starts to increase. Furthermore, this monitoring allows you to retrieve the best model registered over the course of training. This technique is known as Early Stopping and requires you to set up a few things, such as:</p><ul><li><strong>Monitoring model performance</strong>: This monitoring must take place so that the method can activate the stop trigger. Also, this is done on a validation database, which is usually assigned a size of 30% of the training database. Importantly, the validation data is not used in training, so the model will never have seen this validation data when making predictions.</li><li><strong>Trigger to stop training</strong>: From monitoring, the trigger will use a performance metric to know when to stop training. In the simplest case, training is stopped as soon as the performance on the validation data set decreases compared to the performance on the validation data set at the previous training epoch (e.g. an increase in loss). More elaborate triggers may be necessary in practice, as training a neural net is stochastic and can be noisy, so the performance of a model on a validation data set can increase and decrease many times.</li><li><strong>The choice of model to use</strong>: A good practice is to always save the weights of a model if its performance is better than the previous epoch. That way you will always have the model with the best holdout performance.</li></ul><p>The implementation of Early Stopping involves calling it as a callback in the fit method, as shown in the following code.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/54241c317acd3a82424a4ecf702936de/href">https://medium.com/media/54241c317acd3a82424a4ecf702936de/href</a></iframe><h4><strong>Accelerate learning with batch normalization</strong></h4><p>Finally, the last tip we will look at is batch normalization, which is a technique that helps coordinate the updating of multiple layers of the model. It provides an elegant way to repair almost any deep network by normalizing only the mean and variance of each unit to stabilize learning, thus allowing the relationships between units and the non-linear statistics of a single unit to change. Below are some tips on how to use batch normalization.</p><ul><li><strong>Use With Different Network Types</strong>: Batch normalization is a general technique that can be used to normalize the inputs to a layer. It can be used with most network types, such as Multilayer Perceptrons, Convolutional Neural Networks, and Recurrent Neural Networks.</li><li><strong>Probably Use Before the Activation</strong>: Batch normalization may be used on the inputs to the layer before or after the activation function in the previous layer. It may be more appropriate after the activation function for s-shaped functions like the hyperbolic tangent and logistic function. It may be appropriate before the activation function for activations that may result in non-Gaussian distributions like the rectified linear activation function, the modern default for most network types.</li><li><strong>Use Large Learning Rates</strong>: Using batch normalization makes the network more stable during training. This may require a much greater learning rate than standard learning rates, which may further speed up the learning process.</li><li><strong>Less Sensitive to Weight Initialization</strong>: Deep neural networks can be pretty sensitive to the technique used to initialize the weights before training. The stability to training brought by batch normalization can make training deep networks less sensitive to the weight initialization method’s choice.</li><li><strong>Do not Use With Dropout</strong>: Batch normalization offers some regularization effect, reducing generalization error, perhaps no longer requiring dropout for regularization. Further, it may not be good to use batch normalization and dropout in the same network. The reason is that the statistics used to normalize the prior layer’s activations may become noisy given the random dropping out of nodes during the dropout procedure.</li></ul><p>The code below shows an example implementation using the batch normalization technique.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c1a536d67d30dedd7342ed448e8faadc/href">https://medium.com/media/c1a536d67d30dedd7342ed448e8faadc/href</a></iframe><p>You can find more examples and read more about it in <a href="https://github.com/ivanovitchm/ppgeecmachinelearning">Dr. Ivanovitch Silva’s repository</a>.</p><h4>💡Experiments and results</h4><p>First, we will create a model to compare the changes made throughout the notebook. So we create a simple model with one layer containing 100 neurons and with the parameters shown in the following code. You can find this code in section 1.4.4.4 (Creating the model) of notebook 6 (train).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ae960c088ca0e68e2c8957c5897332c6/href">https://medium.com/media/ae960c088ca0e68e2c8957c5897332c6/href</a></iframe><p>The wandb callback allows us to store the information generated during training, as shown in the graphs below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eQNhbKnXIIk__T0I" /></figure><p>In the graphics, it is possible to observe that the validation accuracy became quite noisy, while the error of this base tends to grow, thus showing an indication of an overfitting problem. Thus, for the first comparison test, a model with 50 neurons was compiled with the following sweep configuration (section 1.6.1):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3e89d732f3f1db463c91afd42643bd1f/href">https://medium.com/media/3e89d732f3f1db463c91afd42643bd1f/href</a></iframe><p>With this configuration, we get the results shown in the following graphs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rBXMlZzNz_yOLcckTEUZcA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fMlN7kyGF4ZYxOM-1tmecg.png" /></figure><p>In the graphs, you can see that with batch size equal to 16, the error in the training base decreases faster, although in the test base, the error rises faster. To decrease the training time, the next models will be trained with the EarlyStopping callback. In addition, we will run tests using standardization and normalization of the data (section 1.6.2), as shown earlier. The sweep configuration remained the same and we were able to obtain the following results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VW-_NZpmIIfw2rJ59IrAoQ.png" /></figure><p>It can be seen that EarlyStopping has decreased a lot of training time, since on our didactic basis, the error is tending to increase as epochs go by. Our goal is to improve the generalization of our model, so for the next test, we will use the weight penalty technique and decrease the number of neurons to 10 (section 1.6.3).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W7sbP_LHb55b2bLJkVUB4A.png" /></figure><p>With these changes, we get very interesting results, as we see that after a certain number of epochs, it has remained stable. For the last experiment, we will increase the number of neurons to 15 and use the BatchNormalization technique.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MwedLMCeGB5VkOm9b6_6Eg.png" /></figure><p>At the end of these experiments, we saw the impact that some techniques have on the learning and generalization of a neural network and how you can use them to improve your model. Next, we export the best model obtained and plot the confusion matrix on the test base (notebook 7 — test), thus obtaining the following image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*YwNOUkdRGIc-s_3caeW9DA.png" /></figure><p>In it, it is possible to observe that there were many false positives, which indicates that Lending Club investors would lose a lot of money investing in people that the model said they would pay on time and that, in fact, they would not pay. Thus, comparing with the model made in the first project, the decision tree proved to be a better learning technique for this type of problem. Of course, for other applications, the neural network is a much more powerful tool, but it is still possible to use classical models and obtain good results with them.</p><p>You can make predictions with the model on the internet through the link: <a href="https://credit-risk-nn.herokuapp.com/">https://credit-risk-nn.herokuapp.com/</a></p><h4><strong>Extensions</strong></h4><p>I hope I have helped you understand a little more about deep learning and how to improve your network. You can take my example and create other tests, try new techniques, and so on. Have fun! Here are some extensions you can try:</p><ul><li>Force small weights with weight constraint</li><li>Decouple layers with dropout</li><li>Promote robustness with noise</li><li>Fix exploding gradient with gradient clipping</li></ul><h4>Next Steps in Machine Learning World</h4><p>If you are just starting out in this amazing world, you may be wondering where to go now. Well, I have good news for you: the possibilities are almost endless. If you are interested in learning about text manipulation, you can learn more about NLP (Natural Language Processing) and Recurrent Neural Networks (RNN). On the other hand, if you want to learn about image processing and computer vision, you can go down the path of Convolutional Neural Networks. I hope I can meet you more often and share your projects with me. See you next time!</p><h4><strong>📄 </strong>References</h4><p><strong>Article Github: </strong><a href="https://github.com/Morsinaldo/credit_risk_nn">https://github.com/Morsinaldo/credit_risk_nn</a> <br><strong>Heroku: </strong><a href="https://credit-risk-nn.herokuapp.com/">https://credit-risk-nn.herokuapp.com/</a><strong><br>Dataset: </strong><a href="https://www.kaggle.com/datasets/samaxtech/lending-club-20072011-data">https://www.kaggle.com/datasets/samaxtech/lending-club-20072011-data</a><br><strong>Dr. Ivanovitch Silva&#39;s repository</strong>: <a href="https://github.com/ivanovitchm/ppgeecmachinelearning">https://github.com/ivanovitchm/ppgeecmachinelearning</a><strong><br>Jason Brownlee: </strong>Deep Learning for Natural Language Processing. <a href="https://machinelearningmastery.com/deep-learning-for-nlp/">https://machinelearningmastery.com/deep-learning-for-nlp/</a><br><strong>François Chollet: </strong>Deep learning with Python. <a href="https://www.amazon.com/Deep-Learning-Python-Francois-Chollet/dp/1617294438">https://www.amazon.com/Deep-Learning-Python-Francois-Chollet/dp/1617294438</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=419e5402b4fa" width="1" height="1" alt=""><hr><p><a href="https://blog.devops.dev/learn-how-to-easily-improve-your-neural-network-in-practice-419e5402b4fa">Learn how to easily improve your neural network in practice</a> was originally published in <a href="https://blog.devops.dev/">DevOps.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>