<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by William Aaron on Medium]]></title>
        <description><![CDATA[Stories by William Aaron on Medium]]></description>
        <link>https://medium.com/@waaron.icloud?source=rss-57db7aafd0ad------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*8LP-cdyyhUJTkap-</url>
            <title>Stories by William Aaron on Medium</title>
            <link>https://medium.com/@waaron.icloud?source=rss-57db7aafd0ad------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 13:42:39 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@waaron.icloud/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to Install and Run FTX-Video on Local Computer with GPU(Nvidia) Using ComfyUI]]></title>
            <link>https://medium.com/@waaron.icloud/how-to-install-and-run-ftx-video-on-local-computer-with-gpu-nvidia-using-comfyui-5305e497092c?source=rss-57db7aafd0ad------2</link>
            <guid isPermaLink="false">https://medium.com/p/5305e497092c</guid>
            <dc:creator><![CDATA[William Aaron]]></dc:creator>
            <pubDate>Tue, 17 Dec 2024 12:41:02 GMT</pubDate>
            <atom:updated>2024-12-17T12:41:02.240Z</atom:updated>
            <content:encoded><![CDATA[<p>In this step-by-step guide, I’ll walk you through on how to install and run ftx-video on your local device. Let’s get started!</p><h4>Prerequisites:</h4><ul><li>ComfyUI</li><li>Nvidia GPU</li><li>Git and Git Support for Large Files</li><li>An Archiver(such as 7zip, winrar etc)</li></ul><p>I will assume your device uses an Nvidia GPU and you already have an archiver installed with Git. If you don’t have these installed, you can click on these links to install them:</p><ul><li><a href="https://www.7-zip.org/">For 7zip</a></li><li><a href="https://git-scm.com/downloads/win">For Git</a> and <a href="https://git-lfs.com/">Git Support for Large Files</a></li></ul><h3>Step 1: ComfyUI Installation</h3><p>We will be installing a portable version of ComfyUI. To get started, First visit this <a href="https://github.com/comfyanonymous/ComfyUI/releases">page</a>.</p><p>It will take you to a Github repo where you will download the installation file:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*jaaY4Qq-cGfLZHyg.png" /></figure><p>Once the download is complete, decompress the file with your archiver and move it to the C drive:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*pCVGrPC3FwlpekvQ.png" /></figure><p>The name of the folder is <strong>ComfyUI_windows_portable</strong>. Next, navigate into the folder and execute the file called <strong>run_nvidia_gpu.bat:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*bHVVbnrUeosdZj2n.png" /></figure><p>Executing the file should launch the ComfyUI software in the browser with an IP address:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*0HpzrkjMGUmGcKJE.png" /></figure><h3>Step 2: LTX-model Files Installation in ComfyUI</h3><p>First you need to install ComfyUI-Manager, To do that, click this <a href="https://github.com/ltdrdata/ComfyUI-Manager/raw/main/scripts/install-manager-for-portable-version.bat">link</a></p><p>It will download a bat file which you will save in the base folder of ComfyUI and execute the file:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*QJ0IEiSf3l_wt9Bn.png" /></figure><p>Reopen ComfyUI and make sure you can see the ComfyUI-Manager:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*Qnl4ntbRdIfk02ag.png" /></figure><p>Now you need to install the ComfyUI-LTXVideo node. On the manager menu, click on the Custom Nodes Manager option and search for ComfyUI-LTXVideo and Install:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*kckb3KZNbvPpVBRe.png" /></figure><p>After Installation, restart the program.</p><p>Repeat the same procedure with ComfyUI-VideoHelperSuite, after that you should have these nodes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*j-60l7dtrcwCw8-A.png" /></figure><h3>Step 3: Installation of Model Checkpoints and Files.</h3><p>To continue you need to download the model checkpoints and files. To do so, visit this <a href="https://huggingface.co/Lightricks/LTX-Video/blob/main/ltx-video-2b-v0.9.safetensors">page</a> and download the file called <strong>ltx-video-2b-v0.9.safetensors</strong></p><p>After download, place the file in the directory; <strong>ComfyUI/models/checkpoints</strong></p><p>Then download the file called <strong>t5xxl_fp16.safetensors </strong>from this <a href="https://huggingface.co/Comfy-Org/mochi_preview_repackaged/blob/main/split_files/text_encoders/t5xxl_fp16.safetensors">link</a> and place it in the directory; <strong>ComfyUI/models/text_encoders/</strong></p><h3>Step 4: Installations of Additional Files.</h3><p>Now you need to download some additional resources. First, Open the terminal and follow the order;</p><pre>cd\<br>cd ComfyUI_windows_portable<br>dir<br>cd ComfyUI<br>cd models<br>cd text_encoders<br>git clone https://huggingface.co/PixArt-alpha/PixArt-XL-2-1024-MS</pre><p>Once that’s done, you just need to download the configuration file from this <a href="https://comfyanonymous.github.io/ComfyUI_examples/ltxv/ltxv_text_to_video.json">link</a>.</p><h3>Conclusion</h3><p>Once you’ve completed the previous steps, all you just need to do is restart the program and you’re good to go!!!.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5305e497092c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Anomaly and Fraud Detection in AWS using Machine Learning, Artificial Intelligence and Data…]]></title>
            <link>https://medium.com/@waaron.icloud/anomaly-and-fraud-detection-in-aws-using-machine-learning-artificial-intelligence-and-data-5f3a47d8ef33?source=rss-57db7aafd0ad------2</link>
            <guid isPermaLink="false">https://medium.com/p/5f3a47d8ef33</guid>
            <category><![CDATA[ecs-fargate]]></category>
            <category><![CDATA[fraud-detection]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[William Aaron]]></dc:creator>
            <pubDate>Tue, 19 Nov 2024 22:07:34 GMT</pubDate>
            <atom:updated>2024-11-19T22:07:34.430Z</atom:updated>
            <content:encoded><![CDATA[<h3>Anomaly and Fraud Detection in AWS using Machine Learning, Artificial Intelligence and Data Insights</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d1WEUu1PJCIpp5QhSo_w9A.png" /></figure><h3>Introduction</h3><p>In today’s digital age, the increasing complexity and volume of data present both opportunities and challenges for businesses. One significant challenge is the detection of anomalies and fraudulent activities, which can pose substantial financial and reputational risks. Leveraging the robust suite of tools and services provided by AWS, organizations can harness the power of Machine Learning (ML), Artificial Intelligence (AI), and Data Insights to address these issues effectively.</p><p>This article proposes a Cloud Infrastructure solution to deploy an Anomaly and Fraud detection system on AWS using containerized environments and storage systems. We also proceed to provide a solution to visualize data from the analytics using some managed data visualization services provided by AWS.</p><h3>The Scenario</h3><p>WAARON Finances Inc. is a rapidly growing financial technology company that provides innovative digital banking and payment services to millions of customers worldwide. With the rise in online transactions and digital payments, the company faces increasing risks of fraudulent activities and anomalies within their massive transaction datasets. These risks include unauthorized transactions, account takeovers, and unusual spending patterns that could indicate potential fraud.</p><p>The company aims to leverage AWS’s advanced machine learning, artificial intelligence, and data analytics services to build a solution capable of:</p><ol><li><strong>Identifying Anomalies</strong>: Detecting irregular patterns in transaction data that deviate from typical customer behavior, helping to flag suspicious activities for further investigation.</li><li><strong>Fraud Detection</strong>: Utilizing predictive models to identify potentially fraudulent transactions in real-time, thereby minimizing the financial impact and preventing losses.</li><li><strong>Real-time Monitoring</strong>: Implementing a robust monitoring system to continuously analyze incoming data streams and alert security teams to any potential threats immediately.</li><li><strong>Data Insights</strong>: Gaining deeper insights into customer behavior and transaction trends, allowing for more informed decision-making and strategic planning.</li></ol><h3>The Solution</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d1WEUu1PJCIpp5QhSo_w9A.png" /></figure><p>The Architecture diagram provided above is the proposed Infrastructure for WAARON Finances Inc. This article breaks down each service used in the architecture, why it is used and finally it’s function in the whole architecture workflow.</p><h4>Data Source</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/250/1*pa_pXWASJgdXZlhnFZfnmQ.png" /></figure><p>The data source section is typically a storage solution. This section is the proposed location where WAARON should store all transaction data and CSV files for processing. The S3 bucket serves as the storage location for all csv files and transaction data, while the DynamoDB table stores some transactional metadata.</p><h4>Data Transformation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/433/1*fYkcglAiMLx268MgdybRLg.png" /></figure><p>This section is where the compute and processing happens. The application code is written using the Micro Services pattern with each micro service performing individual functions. Let’s look at each one of the micro service in this project scenario.</p><ol><li><strong>Collector Micro Service</strong>: This is a Java based application that accepts the raw transactions data (most likely compressed) and will extract data from it in a readable and cleaned format. It processes lines of a payment data file, extracts relevant information, processes it, and stores it in a storage service (S3).</li><li><strong>Detector Micro Service: </strong>This is a python based micro service that performs the anomaly detection using Machine Learning techniques. It reads input data from S3 bucket and transforms into embeddings using Standard Scaler and Sentence Transformer, then compares them to model data stored in PostgreSQL. Finally, an enriched dataset with cosine similarity as anomaly signal (between 0 and 1) is stored as<br>output data into S3 bucket.</li><li><strong>Curation Micro Service: </strong>In this project, the data csv contains a column called curation that contains the source IP address of the transaction. What this microservice does is, it extracts all of that data along with some metadata and store in to S3. This data will then be further used for some analytics and visualization data in another section.</li></ol><h3>Data Lake</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/1*qUpL3cFOy7xsFE53qBM8xA.png" /></figure><p>A <strong>data lake</strong> is a centralized repository that allows you to store all your structured and unstructured data at any scale. In this project, S3 and RDS are the two primary managed services used for the data lake, to store data. PostgreSQL in this case is used to store the vectors of historical data with or without fraud.</p><h3>Data Targets</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/227/1*g3JVvXQ661ymbTQjGOS8FA.png" /></figure><p>This section is where finalized data stored in S3 is queried with Amazon Athena and is further visualized using Amazon QuickSight. Amazon Athena provides features to query directly from S3 using SQL statements these query data is then converted into appealing dashboards to analyze for Fraud in certain accounts.</p><h3>Management</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/711/1*cT_KuoudS7WcMFVu9Hi6TQ.png" /></figure><h4>GitHub Actions</h4><p>DevOps practices is also incorporated into the architecture specifically CICD pipeline using GitHub Actions to pick changes of the application code and automatically deploy new versions of the ECS Task Definition. The pipeline also builds up new application docker image that is stored in ECR.</p><h4>AWS ECR</h4><p>Elastic Container Registry is a managed service on AWS to store different versions of docker images. In this project, ECR is used to store multiple versions of the python and Java micro services.</p><h4>IAM Roles</h4><p>To ensure security practices in the application, IAM roles to only allow least privileged access are incorporated into every service ensuring best security practices are followed.</p><h4>Networking</h4><p>VPC, Subnets, and Security groups are used for the networking bit of the architecture, a NAT gateway is also configured to ensure the the ECS tasks are in the private subnet without inbound internet connection traffic.</p><h3>Domain and API Interface</h3><p>Finally, an API interface is provided to trigger the fraud detection analysis upon HTTP request that is sent to it. The Interface is built with API Gateway which is directly integrated with ECS with the aid of Amazon CloudMap services to load balancer the ECS tasks behind the API gateway.</p><p>Route 53 in this architecture is used to map a custom domain to the API gateway invoke URL.</p><h3>Conclusion</h3><p>In an era where digital transactions are rapidly increasing, the ability to effectively detect and prevent anomalies and fraud is crucial for financial institutions like WAARON Finances Inc. By leveraging AWS’s robust suite of services, the proposed solution provides a comprehensive and scalable approach to identifying and mitigating fraudulent activities in real-time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5f3a47d8ef33" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Architecting a Video Processing Application with Machine Learning Features on AWS]]></title>
            <link>https://medium.com/@waaron.icloud/architecting-a-video-processing-application-with-machine-learning-features-on-aws-70da50c4fbc5?source=rss-57db7aafd0ad------2</link>
            <guid isPermaLink="false">https://medium.com/p/70da50c4fbc5</guid>
            <category><![CDATA[amazon-rekognition]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[serverless-architecture]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[William Aaron]]></dc:creator>
            <pubDate>Sun, 17 Nov 2024 01:17:01 GMT</pubDate>
            <atom:updated>2024-11-17T01:22:24.192Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8JOblm5jbEGcouEWam4HJw.png" /><figcaption>Project Diagram</figcaption></figure><p>Applications with Machine Learning capabilities or AI use cases is now the new big thing with the rise in Big Data, GenAI, and other new technologies. AWS captures this market opportunity, and provides us with managed services to streamline and reduce heavy compute tasks, furthermore giving Cloud Infrastructure Engineers more time to build.</p><p>In this article, we will look at a project scenario with AWS services chosen for the solution and finally, how it can be applied. We will also look into the functionalities of these AWS services and how they’re put together.</p><h3>The Scenario</h3><p>A thriving software company named WAARON LLC looking to enhance its media capabilities with advanced video processing solutions. WAARON LLC requires a robust system that can seamlessly handle various tasks related to video content analysis. The desired solution needs to be capable of tagging videos and identifying specific elements within them, such as objects, scenes, and activities.</p><p>Additionally, the system should efficiently frame the videos into sections for better identification and analysis. An essential feature of the solution is its ability to recognize celebrities appearing in the video content. Furthermore, WAARON LLC seeks to integrate brand recognition capabilities to identify logos or text within the videos, ensuring comprehensive and detailed media analysis.</p><p>By leveraging AWS’s powerful suite of tools and AI enabled services, WAARON LLC aims to build a sophisticated video analysis solution that meets all these requirements, paving the way for more innovative media applications and enhanced user experiences.</p><h3>The Solution</h3><p>The AWS infrastructure chosen for WAARON LLC involves a blend of serverless and Containerized solution, The serverless lambda functions will Serve as the API interface to perform functions such as Executing the Analysis, Querying through Analysis Results, and finally, searching through all the Analysis that has been performed to get their JobId. Let’s have a look at the Architecture Diagram given below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*8JOblm5jbEGcouEWam4HJw.png" /></figure><p>We will break the Architecture diagram into 5 sections and analyze each part of it and it’s function, and further explain how data is passed from each section to another and why each AWS service is choosen.</p><h4>API &amp; Domain</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/190/1*3RZAgKuLFoWZBIw4Cg8Xhg.png" /></figure><p><strong>Route 53</strong> is AWS’s service for domain name registrations and record mappings this will allow WAARON LLC to map their custom domain to the API gateway endpoint that will be provided.</p><p><strong>API Gateway </strong>is a fully managed Service provided by AWS that enables developers to create, publish, and maintain APIs at any scale. It acts as a bridge between client applications and backend services, allowing seamless communication and data exchange.</p><p>Using API gateway will ensure that WAARON LLC’S API can handle any amount of traffic with automatic scaling, Security with IAM roles and API keys, Monitoring and Analytics, and finally, a cost effective solution because InnovateSoft will only pay for the amount of data transferred, and the amount of API calls received.</p><h4>Lambda Functions for API Endpoints</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/204/1*O-PuQvSnI--Hovc5eXwK6Q.png" /></figure><p>The three lambda functions in this sections are the three integrations that will be added to the API gateway to perform certain actions upon invocation. Each action is explained below:</p><p><strong>Start Analysis</strong>: The start analysis lambda function upon invocation, takes the video file name from the request and input sends it to Amazon Media Convert as an S3 bucket object name. This means that the file name added to the request sent to this lambda function must already exist in an input s3 bucket.</p><p>It then proceeds to collect the video frames from Amazon Media convert and store into another S3 bucket <strong>output bucket. </strong>This video frames can then be used for the analysis in further steps. The start analysis function also gives the video analysis a jobId and store it’s info into a DynamoDB table and create an index for it in Amazon Opensearch.</p><p><strong>Query</strong>: This function simply collects the search analysis results from dynamodb and return it as a JSON response. This analysis results includes tags identified in the video such as object identification, It also returns a list of brands identified from either brand logo or brand texts.</p><p><strong>Search: </strong>The search function is a lambda function that integrates with Amazon Opensearch to search through the analysis results. For example, say you want to search for a video where a cat appear, or a video that has a Mercedes Benz car. The function will search through the result of all anaylsis and respond back with video analysis results that match the query.</p><h4>Media Convertion</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/272/1*BphPq0fgd4yqjz7WOgQWXw.png" /></figure><p><strong>AWS Environmental MediaConvert </strong>is a powerful fully managed service that offers video transcoding and processing. It allows users to convert vidoes into different formats. In this project, we use MediaConvert to extract all the frames of the video and the audio of the provided video. These frames are then used to perform analysis frame by frame.</p><p>The service is a fully managed service and the billing structure is a pay-as-you-go pricing, ensuring cost efficiency based on the duration and complexity of the video processed.</p><p><strong>MediaConvert Checker </strong>There’s an event bridge rule that is integrated with a lambda function to check the status of the video conversion. If the video conversion is successful, it then proceeds to start the analysis and send notifications through an SNS topic about the status of the job. Else, only send a notification that the job was unsuccessful.</p><h4>The Analysis</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/176/1*Q2YfJ6g6u-JXUY1_qF71zg.png" /></figure><p>This section contains ECS FARGATE tasks which contains containerized codebase to perform analysis, these ECS tasks are APIs written by the developers. These APIs in turn send requests to Amazon Rekognition to perform analysis.</p><p>These APIs, check the output bucket where the video frames are, take each frame of the video and send to amazon recognition for analysis. It then saves the analysis results into a DynamoDB table. Each API microservice written by the developers perform individual tasks for every use case.</p><h4>Storage and Emails</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/291/1*rIzUU1XWbTBUKJ39A2x7xA.png" /></figure><p>This last section contains an SNS topic to send email notifications to about the status of the analysis, the DynamoDB table provisioned to store analysis results and finally Opensearch used to perform powerful high-end search queries which in this project is used to query analysis results with matching criteria.</p><h3>Conclusion</h3><p>The integration of machine learning and AI capabilities into video processing applications is not just a trend, but a significant leap forward in how media content is managed and utilized. WAARON’s comprehensive video analysis solution showcases the potential of AWS’s powerful suite of tools and managed services. By leveraging a blend of serverless computing and containerized solutions, InnovateSoft ensures that its system is both scalable, efficient and Cost friendly.</p><p>WAARON not only meets its current needs but also positions itself at the forefront of media innovation, ready to tackle future challenges and opportunities. The potential for enhancing user experiences, improving content management, and driving forward the capabilities of media applications is immense, making this a truly transformative approach to video processing.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=70da50c4fbc5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Set Up Azure Pipelines Self Hosted Runners using EC2 Instances]]></title>
            <link>https://medium.com/@waaron.icloud/set-up-azure-pipelines-self-hosted-runners-using-ec2-instances-355471d65224?source=rss-57db7aafd0ad------2</link>
            <guid isPermaLink="false">https://medium.com/p/355471d65224</guid>
            <category><![CDATA[azure-devops-pipeline]]></category>
            <category><![CDATA[azure-pipelines]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[ec2]]></category>
            <category><![CDATA[cicd]]></category>
            <dc:creator><![CDATA[William Aaron]]></dc:creator>
            <pubDate>Sun, 01 Sep 2024 23:08:42 GMT</pubDate>
            <atom:updated>2024-09-01T23:08:42.265Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*k6gNt__KWt02xmx6CHAMtg.png" /></figure><h3>Introduction</h3><p>Repository URL: <a href="https://github.com/devopslord/azure-cicd-pipeline.git">https://github.com/devopslord/azure-cicd-pipeline.git</a></p><p>CICD is now a requirement in modern day software application deployments. Automating build processes, testing, vulnerability scanning and even deployment is now a game changer in application deployments and DevOps implementations. But first, what is CICD?</p><p>CICD, which is Continuous Integration and Continuous Deployment, are set of practices that automates building, testing, and publishing software applications. It is a key part in the DevOps methodology and plays a key part in shortening the software development lifecycle. There are many CICD tools that are used to implement these practices, tools such as Jenkins, GitHub Actions, Circle CI, Azure Pipelines, etc.</p><p>In this article, we will focus on Azure’s tool for CICD. The article walks you though how to set up a self hosted runner for Azure pipeline using EC2 Instance in AWS.</p><h4>Prerequisites</h4><p>To follow along with this article, you’re expected to meet the following requirements:</p><ul><li>Azure Organization Account</li><li>AWS Account</li><li>Terraform Knowledge</li></ul><p>The article sets up the EC2 instance on AWS using terraform and proceeds to execute scripts that will install all required scripts to add the EC2 instance as an agent in Azure pipelines. Let’s proceed to create the Azure DevOps project.</p><h4>Create Azure DevOps Project</h4><p>In your Azure DevOps console, select your organization or create a new one an create a new project in the organization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*bsliDglI0WugOEC7K2WDSg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eLiQu-KPwGBBrC0W-M1OFw.png" /></figure><p>Once the Azure DevOps project is created, let’s spin up the EC2 instance on AWS. We will then add the instance as an agent in Azure DevOps project agent pools.</p><h4>Create Instance On AWS</h4><p>To create the instance on AWS, we will be using terraform to spin up an instance. The terraform configuration will create the instance on the default subnet, and will generate a key pair on your local machine with which we will use to SSH into the instance and pass in commands to install dependencies we will need.</p><p>This article is provided with a repository that contains all necessary terraform configuration files to spin up an ec2 instance on AWS using terraform. Clone the repository on your local machine an execute the commands on a local terminal in the repository directory:</p><pre>terraform init<br>terraform validate <br>terraform apply --auto-approve</pre><p>Once terraform completes you should get an output with the instance public IP.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/559/1*K76SEnXxwis5Xq_YFNRJjg.png" /></figure><p>An SSH key will also be generated for you in the project’s directory:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/205/1*CLeSsehVs97z9ACaiKQAJw.png" /></figure><h4>Generate Azure Organization Personal Access Token</h4><p>To move forward, we will need to generate a personal access token to allow the EC2 instance to authenticate to the azure devops organization. On your organization’s UI in the URL format `<a href="https://dev.azure.com/sherif0763/">https://dev.azure.com/&lt;organization name&gt;/</a>`, Click on <strong>User Settings</strong> and click on personal access token an generate one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tgb-JYdQYUdRDoElkjYIkA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/409/1*oQsNXC0a8_5R7jcSBLuR4Q.png" /></figure><p>Give the token a name and grant full access. Feel free to define a custom scope with defined access.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*QrV8vk5pj_suzpnRmLgnRw.png" /></figure><p>Copy the token and keep in a safe place, we will need the token when setting up the ec2 instance as an azure pipeline agent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xM8Pt6WOW1PFam43KGtbog.png" /></figure><h4>Add a New Agent</h4><p>To add the agent to azure pipeline, we will proceed to SSH into the EC2 instance to download the agent configuration an set up the agent.</p><p>SSH into the EC2 instance created using the terraform script using the command below (ensure you edit the public IP):</p><pre>ssh -i .\id_RSA ubuntu@&lt;public IP&gt;</pre><p>Download the agent to the instance using the command below:</p><pre>wget https://vstsagentpackage.azureedge.net/agent/3.243.1/vsts-agent-linux-x64-3.243.1.tar.gz</pre><p>Create a dedicated folder to install the agent:</p><pre>mkdir myagent &amp;&amp; cd myagent</pre><p>Extract the agent from the tar file downloaded previously using the command below:</p><pre>tar zxvf /home/ubuntu/vsts-agent-linux-x64-3.243.1.tar.gz</pre><p>Let’s proceed to configure the Agent to point to our azure devops organization. In /myagent folder where we extracted the agent tar file, you should file a config.sh file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*ww0V3tM7J9_6nYOwPHvaiw.png" /></figure><p>Execute the config file and fill in each step to configure the agent. execute the command to execute the sh file</p><pre>./config.sh</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*djfvZoatkgydimlUYvQn3g.png" /></figure><p>For server URL, enter your azure devops organization URL with the format <a href="https://dev.azure.com/%7Byour-organization%7D">https://dev.azure.com/{your-organization}</a></p><p>For authentication type, press enter to select PAT (Personal Access Token). Proceed to paste in the token we generated in the previous step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/1*IwAlD6mT3X56LoV9mQF3wA.png" /></figure><p>For agent pool, press enter to select the default agent pool. Agent pools are a way to group instances in your azure DevOps organization. A good example of how you can use agent pools is when you want to separately group agents with GPU from agents without GPU, or group windows agents from Linux agents.</p><p>Finally, enter any agent name of your choice in my case, I named mine ec2-us-east-1. Press enter to finalize the process.</p><p>To confirm that the agent got added, on your Azure DevOps project dashboard, click on project settings</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/266/1*Lonh_Un3lis-WMemnh4gPg.png" /></figure><p>Under pipelines on the left panel, click on agent pools. Select the default pool and click on the Agents tab you should find the ec2 instance with the name we configured.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HWj5ndqyznic8vkNFx9eew.png" /></figure><h4>Activate the Agent</h4><p>You’ll notice that the agent is in an offline state. To activate the agent to be back online, you can simply execute the run.sh script inside the ec2 instance in the preconfigured folder we installed the agent.</p><p>Using the script works fine, but once you close the terminal or stop the script session, the runner goes back offline. To tackle this, we will activate the script to run as a systemd service on the background. To do this, in the folder we installed the agent, in my case /home/ubuntu/myagent, execute the command below</p><pre>sudo ./svc.sh install ubuntu</pre><p>Note that ubuntu is the username and root user cannot be used. To start the service, execute the command below:</p><pre>sudo ./svc.sh start</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5hB7T9zanUshHWvUDAnZUw.png" /></figure><p>You should see that the runner is actively online in Azure DevOps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o0Npg71ABpjlAmyLW8ruvg.png" /></figure><p>You stop the runner using the command below:</p><pre>sudo ./svc.sh stop</pre><h3>Conclusion</h3><p>In this article, we went through how to spin up an EC2 instance on AWS using terraform and then proceeded to make the instance an agent to pick up azure DevOps pipelines jobs. In the next article, we will look through how to set up a CICD pipeline in Azure DevOps that will build and push image to ECR and deploy the new image version to ECS tasks.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=355471d65224" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MLOps Engineer: Productionizing Machine Learning Workloads.Part 3]]></title>
            <link>https://medium.com/@waaron.icloud/mlops-engineer-productionizing-machine-learning-workloads-part-3-e0f028861a4c?source=rss-57db7aafd0ad------2</link>
            <guid isPermaLink="false">https://medium.com/p/e0f028861a4c</guid>
            <dc:creator><![CDATA[William Aaron]]></dc:creator>
            <pubDate>Sun, 14 Jan 2024 21:10:10 GMT</pubDate>
            <atom:updated>2024-01-14T21:10:10.359Z</atom:updated>
            <content:encoded><![CDATA[<p>Link to Part 1 : <a href="https://medium.com/@waaron.icloud/mlops-engineer-productionizing-machine-learning-workloads-1eccb513ee0a">https://medium.com/@waaron.icloud/mlops-engineer-productionizing-machine-learning-workloads-1eccb513ee0a</a></p><p>Link to Part 2: <a href="https://medium.com/@waaron.icloud/mlops-engineer-productionizing-machine-learning-workloads-part-2-9156c7a58258">https://medium.com/@waaron.icloud/mlops-engineer-productionizing-machine-learning-workloads-1eccb513ee0a</a></p><h3>Part 3.</h3><p><strong>Setting up a Blob Storage account datastore using the Azure Machine Learning CLI.</strong></p><ol><li>Create a yml file blobstore.yml.</li></ol><p>The Account key and name is located in your <strong>azure storage account.</strong></p><pre>$schema: https://azuremlschemas.azureedge.net/latest/azureBlob.schema.json<br>name: blobe_storage<br>type: azure_blob<br>description: Datastore pointing to a blob container.<br>account_name: amlwsstoxxxxxxxxxxxxxxxxxxxx<br>container_name: azxxx<br>credentials:<br>  account_key: xlaGrzO7Gxh5cAiPI0yVOeMqLFxxxxxxxxxxxxxxxxxxxxxxxx<br></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iBzjYrDjdGgnEX2RPtJ7sw.png" /></figure><p>2. Execute the following in your terminal.</p><p><em>az ml datastore create — file blobstore.yml — resource-group aml-dev-rg — workspace-name aml-ws</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iBzjYrDjdGgnEX2RPtJ7sw.png" /></figure><p>3. Confirm that the blobstore is created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TZtBdScmnvhpx4QcfB4tGg.png" /></figure><h3><strong>Setting up data asset using the Python SDK.</strong></h3><ol><li>Create a folder with two files titanic.csv and yml file blobstore.yml.</li><li>Create my_data/env-mltable.yml,my_data/MLTable.yml and my_data/titanic.csv</li></ol><p>The Account key and name is located in your <strong>azure storage account.</strong></p><pre># Enter details of your AML workspace<br>subscription_id = &quot;39ee2a0a-xxxxxxxxxxxxxxxxxxx&quot;<br>resource_group = &quot;aml-dev-rg&quot;<br>workspace = &quot;aml-ws&quot;<br><br># get a handle to the workspace<br>ml_client = MLClient(<br>    DefaultAzureCredential(), subscription_id, resource_group, workspace<br>)<br><br>from azure.ai.ml.entities import Data<br>from azure.ai.ml.constants import AssetTypes<br><br># my_path must point to folder containing MLTable artifact (MLTable file + data<br># Supported paths include:<br># local: &#39;./&lt;path&gt;&#39;<br># blob:  &#39;https://&lt;account_name&gt;.blob.core.windows.net/&lt;container_name&gt;/&lt;path&gt;&#39;<br># ADLS gen2: &#39;abfss://&lt;file_system&gt;@&lt;account_name&gt;.dfs.core.windows.net/&lt;path&gt;/&#39;<br># Datastore: &#39;azureml://datastores/&lt;data_store_name&gt;/paths/&lt;path&gt;&#39;<br><br>my_path = &#39;./my_data/&#39;<br><br>my_data = Data(<br>    path=my_path,<br>    type=AssetTypes.MLTABLE,<br>    description=&quot;description&quot;,<br>    name=&quot;titanic-mltable&quot;,<br>    version=&#39;1&#39;<br>)<br><br>ml_client.data.create_or_update(my_data)</pre><pre>#./my_data/env-mltable.yml<br>name: mltable-env<br>channels:<br>  - conda-forge<br>dependencies:<br>  - python=3.8<br>  - pip=21.2.4<br>  - pip:<br>    - mltable<br>    - pandas==1.3.0</pre><pre>#./my_data/MLTable.yml<br>type: mltable<br><br>paths:<br>    - file: ./titanic.csv<br><br>transformations:<br>    - read_delimited:<br>        encoding: ascii<br>        header: all_files_same_headers<br>        delimiter: &#39;,&#39;</pre><p>./my_data/titanic.csv <a href="https://raw.githubusercontent.com/devopslord/aml/main/part-3/my_data/titanic.csv">https://raw.githubusercontent.com/devopslord/aml/main/part-3/my_data/titanic.csv</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/1*uFTaVx6S1L8yxchW1YFovQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/1*9P8GXBziTjUz9b2P8xnhRg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EmJA8sZXujPeEm26A2WMbQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1hdX-F36qSA-8OfcT3YAJg.png" /></figure><p>3. Converting MLTables to Pandas data frames</p><pre>import mltable<br>tbl = mltable.load(uri=&quot;./my_data&quot;)<br>df = tbl.to_pandas_dataframe()<br>df</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fwWX5R_r_mUVCH2sxm7zIg.png" /></figure><p>4. Read data in a job</p><p>A job encompasses a Python script, ranging from straightforward data processing to intricate code for model development. It involves a Bash command that outlines the tasks, inputs, and outputs of the job. Additionally, a Docker environment is specified, detailing the runtime libraries necessary for executing the job. The job runs within a Docker container on a designated compute resource. Data is typically passed to an Azure Machine Learning job through datasets, with the Python script inside the job interacting with these datasets.</p><p>The Steps needed to create a job that takes dasets as inputs.</p><p>i. Create a AML environment to process and train your model.</p><pre>from azure.ai.ml.entities import Environment<br><br>env_docker_conda = Environment(<br>    image=&quot;mcr.microsoft.com/azureml/openmpi3.1.2-ubuntu18.04&quot;,<br>    conda_file=&quot;./my_data/env-mltable.yml&quot;,<br>    name=&quot;mltable&quot;,<br>    description=&quot;Environment created for consuming MLTable.&quot;,<br>)<br><br>ml_client.environments.create_or_update(env_docker_conda)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PWI7OTnPsIP8QIARR0Z57Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/782/1*v9IKhX5MIasP9JL3a5QC5A.png" /></figure><p>ii. Basic Python code that receives an input dataset, transforms it into a pandas data frame, and subsequently displays the data frame. e.g read_data.py</p><pre>import argparse<br>import mltable<br><br>parser = argparse.ArgumentParser()<br>parser.add_argument(&quot;--input_data&quot;, type=str)<br>args = parser.parse_args()<br><br>tbl = mltable.load(args.input_data)<br>df = tbl.to_pandas_dataframe()<br>print(df)</pre><p>iii. Create an AML compute cluster where the Machine Learning containerized job will be submitted.</p><pre>from azure.ai.ml.entities import ComputeInstance, AmlCompute<br><br>compute_cluster = AmlCompute(<br>    name=&quot;cpu-cluster&quot;,<br>    type=&quot;amlcompute&quot;,<br>    size=&quot;STANDARD_DS3_v2&quot;,<br>    location=&quot;eastus&quot;,<br>    min_instances=0,<br>    max_instances=2,<br>    idle_time_before_scale_down=120,<br>)<br>ml_client.begin_create_or_update(compute_cluster)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*leZ7xLqPIBGETGoaPulRdQ.png" /></figure><p>4. We now possess all the essential components to assemble an AML job and initiate its execution. The Python script responsible for crafting an AML job is named “job.” This job essentially encapsulates a Docker container housing your Python code in read_data.py. This container takes your pre-existing input dataset and is dispatched to the compute cluster we created earlier.</p><pre>from azure.ai.ml import command<br>from azure.ai.ml.entities import Data<br>from azure.ai.ml import Input<br>from azure.ai.ml.constants import AssetTypes<br><br><br># Possible Paths for Data:<br># Blob: https://&lt;account_name&gt;.blob.core.windows.net/&lt;container_name&gt;/&lt;folder&gt;/&lt;file&gt;<br># Datastore: azureml://datastores/paths/&lt;folder&gt;/&lt;file&gt;<br># Data Asset: azureml:&lt;my_data&gt;:&lt;version&gt;<br><br><br># for example you can use either one of the following paths:<br># inputs = {&quot;input_data&quot;: Input(type=AssetTypes.MLTABLE, path=&quot;./my_data/&quot;)}<br># or<br>inputs = {&quot;input_data&quot;: Input(type=AssetTypes.MLTABLE, path=&quot;azureml:titanic-mltable:1&quot;)}<br><br>job = command(<br>    code=&quot;./my_data&quot;,  # local path where the code is stored<br>    command=&quot;python read_data.py --input_data ${{inputs.input_data}}&quot;,<br>    inputs=inputs,<br>    environment=env_docker_conda,<br>    compute=&quot;cpu-cluster&quot;,<br>)<br><br># submit the command<br>returned_job = ml_client.jobs.create_or_update(job)<br># get a URL for the status of the job<br>returned_job.services[&quot;Studio&quot;].endpoint</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rY55rWnz5TB9Y8cRJhykWQ.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0f028861a4c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MLOps Engineer: Productionizing Machine Learning Workloads.Part 2]]></title>
            <link>https://medium.com/@waaron.icloud/mlops-engineer-productionizing-machine-learning-workloads-part-2-9156c7a58258?source=rss-57db7aafd0ad------2</link>
            <guid isPermaLink="false">https://medium.com/p/9156c7a58258</guid>
            <dc:creator><![CDATA[William Aaron]]></dc:creator>
            <pubDate>Mon, 01 Jan 2024 20:28:22 GMT</pubDate>
            <atom:updated>2024-01-01T20:28:22.493Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong><em>Part 2.</em></strong></h4><p>Link to Part 1 : <a href="https://medium.com/@waaron.icloud/mlops-engineer-productionizing-machine-learning-workloads-1eccb513ee0a">https://medium.com/@waaron.icloud/mlops-engineer-productionizing-machine-learning-workloads-1eccb513ee0a</a></p><h4><strong>Creating a Compute for Code Development</strong></h4><p>In this segment, you’ll establish a compute instance to kickstart your development journey. Each subsequent section will guide you through the process of creating these essential resources within your Azure Machine Learning Service (AMLS) workspace, employing MLOps as Code.</p><p>On your Terminal, execute the following.</p><p>az ml compute create \<br> — name amlcomputeinstance01 \<br> — size STANDARD_D3_V2 \<br> — type ComputeInstance \<br> — resource-group aml-dev-rg \<br> — workspace-name aml-ws</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jY82yIBllvpp6YzfMe5U4g.png" /></figure><h3><strong>Integrating AMLS with VSCODE</strong></h3><p>Once your Compute Instance VM for computing is set up, you can utilize it for coding in either R or Python. Notably, you have the flexibility to code in Jupyter, JupyterLab, Visual Studio Code (VSCode), or a terminal. Jupyter and JupyterLab serve as integrated development environments (IDEs) for Python coding, while VS Code, recommended by Microsoft, supports scripting in both R and Python, along with numerous other languages.</p><p>In this segment, we’ll kick off by Integrating your VSCODE with your AMLS by establishing a connection to your AMLS workspace.</p><ol><li>Download and install VS Code at <a href="https://code.visualstudio.com/download.">https://code.visualstudio.com/download.</a></li><li>From within VS Code, click on the <strong>EXTENSIONS </strong>icon (<em>Ctrl + Shift + x</em>), search for Azure Machine Learning, and select <strong>Install</strong>:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P083BQk5wUPC-W1UJxDJJw.png" /></figure><p>3. Log in to your Azure account by pressing Ctrl + Shift + P and entering the following command:<br> &gt;Azure: Sign In<br> A new browser window will open for you to provide your credentials and complete the sign-in process.</p><p>5. Select your default workspace through the command palette (Ctrl + Shift + P) and enter:<br> &gt;Azure ML: Set Default Workspace<br> Follow the prompts to choose your subscription and workspace settings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kRGYIM2px_lihLkBxSosVg.png" /></figure><p>6. Navigate to the Azure icon (Shift + Alt + A), and access the MACHINE LEARNING section.<br>7. In the Machine Learning section of the Azure icon, right-click as depicted below and choose Connect for our compute instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*nDWiEu4pYaD0Qx592CioJA.png" /></figure><p>8. Upon execution, a fresh instance of VS Code will be initiated on your local machine. Within this new VS Code instance, you’ll observe the user directory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s1W_FnMnv_Gv8raj39roUw.png" /></figure><p>9. Create a new file devcode.ipynb</p><p>10. Choose your Python interpreter from the top-right corner by clicking on “Select Kernel.” Opt for the “azureml_py310_sdkv2” kernel.</p><p>11. Once you’ve chosen the interpreter, proceed to create a print statement.</p><p>import os</p><p>directory_path = os.getcwd()</p><p>print(“My current directory is :” + directory_path)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DT9l6czlnPiTF66MP-xP7Q.png" /></figure><p>To run this code, click the play button to the left of the cell (Ctrl + Enter). Keep in mind that the provided Python code will display the current working directory. The execution occurs on the compute instance, as evidenced by the printed directory path.</p><p>Saving the notebook within VS Code will store the notebook in your Azure Machine Learning Service (AMLS) workspace.</p><p>In this segment, you’ve installed VS Code, the AML VS Code extension, and established a connection to your compute instance within your Azure Machine Learning Service (AMLS) workspace for code execution. VS Code offers IntelliSense, code running and debugging capabilities, and seamless Git integration. The amalgamation of these features, coupled with integration into your AMLS workspace, positions VS Code as the optimal choice for development.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9156c7a58258" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MLOps Engineer: Productionizing Machine Learning Workloads.]]></title>
            <link>https://medium.com/@waaron.icloud/mlops-engineer-productionizing-machine-learning-workloads-1eccb513ee0a?source=rss-57db7aafd0ad------2</link>
            <guid isPermaLink="false">https://medium.com/p/1eccb513ee0a</guid>
            <dc:creator><![CDATA[William Aaron]]></dc:creator>
            <pubDate>Mon, 01 Jan 2024 19:05:18 GMT</pubDate>
            <atom:updated>2024-01-01T19:05:18.476Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Part 1</strong>.</h3><p><strong>Introduction:</strong></p><p>In the dynamic landscape of today’s technological revolution, businesses are increasingly relying on data-driven decision-making processes to gain a competitive edge. With the surge in artificial intelligence and machine learning applications, the intersection of data science and operations has given rise to a transformative discipline known as MLOps, or Machine Learning Operations.</p><p>MLOps represents a paradigm shift, acknowledging that the journey from developing a machine learning model to deploying it into real-world applications is not a linear path. Traditionally, data scientists and operations teams functioned in silos, often leading to challenges in model deployment, monitoring, and overall efficiency. MLOps emerges as the solution, fostering collaboration and efficiency across the entire machine learning lifecycle.</p><p>At its core, MLOps is a set of practices, principles, and tools that streamline and automate the end-to-end process of deploying, managing, and scaling machine learning models in production. It addresses the complexities associated with transitioning from experimentation in the development phase to the seamless integration of models into operational systems.</p><p>Key Components of MLOps:</p><ol><li>Collaboration and Communication: MLOps emphasizes the need for strong collaboration between data scientists, developers, and operations teams. Effective communication ensures a shared understanding of project goals, requirements, and potential challenges, fostering a collaborative environment.</li><li>Automation: Automation lies at the heart of MLOps, enabling the automated deployment and scaling of machine learning models. Continuous integration and continuous deployment (CI/CD) pipelines are employed to automate testing, validation, and deployment processes, reducing manual intervention and minimizing errors.</li><li>Model Monitoring and Management: MLOps ensures that machine learning models are continuously monitored in real-time once deployed. This includes tracking model performance, detecting anomalies, and managing model versioning to facilitate updates and improvements.</li><li>Scalability: As organizations scale their machine learning initiatives, MLOps provides the infrastructure and practices to efficiently handle increased workloads. Scalability is crucial to meet the demands of growing datasets and evolving business requirements.</li><li>Security and Governance: MLOps incorporates robust security measures and governance protocols to protect sensitive data and ensure compliance with regulatory standards. This is particularly important as machine learning applications increasingly handle valuable and confidential information.</li></ol><p>In the following sections of this article, I will show you a Hands-On Approach to MLOps in Azure Machine learning. I am a fan of Everything as code, We use the terminal and later a complete automation of the processes. From fostering collaboration to embracing automation and scalability, MLOps represents a pivotal advancement in the integration of machine learning into the operational fabric of businesses, driving innovation and efficiency.</p><p><strong>Setting up Machine Learning Environment:</strong></p><ol><li>Create your Microsoft Azure Cloud. Goto <a href="https://azure.microsoft.com/">https://azure.microsoft.com</a></li><li>Install the Azure CLI from <a href="https://docs.microsoft.com/en-us/cli/azure/install-azure-cli">https://docs.microsoft.com/en-us/cli/azure/install-azure-cli</a>.</li><li>Locate your subscription ID in the Azure portal by searching for “Subscriptions” in the box. Retrieve the Subscription ID for the desired subscription.</li><li>On you <strong>Terminal. E</strong>xecute the following steps</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tmBgqe_CdIP0BOrleMpynw.png" /></figure><p>You have now established an AMLS workspace using the Azure CLI ml extension.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1eccb513ee0a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bitbucket to Gihub Repositories Migration: A GitHub Workflow/Bash Script Solution.]]></title>
            <link>https://medium.com/@waaron.icloud/bitbucket-to-gihub-repositories-migration-a-github-workflow-bash-script-solution-3e6992331107?source=rss-57db7aafd0ad------2</link>
            <guid isPermaLink="false">https://medium.com/p/3e6992331107</guid>
            <dc:creator><![CDATA[William Aaron]]></dc:creator>
            <pubDate>Sun, 17 Dec 2023 16:58:26 GMT</pubDate>
            <atom:updated>2023-12-17T16:58:26.105Z</atom:updated>
            <content:encoded><![CDATA[<p># Migrating Bitbucket Repositories to GitHub: A Step-by-Step Guide</p><p>Migrating repositories from Bitbucket to GitHub can be a streamlined process when using automation. In this guide, we’ll walk you through the steps to migrate multiple Bitbucket repositories listed in a text file to GitHub. The solution uses a GitHub Actions workflow along with a Bash script to automate the migration.</p><p>## Prerequisites:</p><p>1. **GitHub Account:** Ensure you have a GitHub account. If not, create one at [GitHub](<a href="https://github.com/">https://github.com/</a>).</p><p>2. **Access to Bitbucket Repositories:** Have access to the Bitbucket repositories you want to migrate.</p><p>3. **GitHub Token:** Generate a GitHub token with sufficient permissions. You can create one in your GitHub account under “Settings” -&gt; “Developer settings” -&gt; “Personal access tokens.”</p><p>4. **Secrets in GitHub Repository:**<br> — Add the GitHub token as a secret in your GitHub repository with the name `GITHUB_TOKEN`.<br> — Add Bitbucket credentials (username and token) as secrets (`BITBUCKET_USERNAME` and `BITBUCKET_TOKEN`).</p><p>## GitHub Actions Workflow:</p><p>Below is a GitHub Actions workflow YAML that automates the migration. Save this in a file, e.g., `migrate.yml` in the `.github/workflows/` directory of your repository.</p><p>```yaml<br>name: Bitbucket to GitHub Repositories Migration</p><p>on:<br> workflow_dispatch:</p><p>env:<br> Github_ORG_NAME: new-github-org</p><p>jobs:<br> Bitbucket_Github_Migration:<br> runs-on: ubuntu-latest<br> <br> steps:<br> — name: Checkout Code<br> uses: actions/checkout@v3</p><p>- name: Execute Migration Script<br> run: |<br> #!/bin/bash</p><p># Logging<br> DATE=$(date +”%Y-%m-%d-%T”)<br> SCRIPT_DIR=$(cd “$(dirname “${BASH_SOURCE[0]}”)” &amp;&amp; pwd)<br> SCRIPT_NAME=”BBReposMigrationToGitHub”<br> LOGFILE=$SCRIPT_DIR/$SCRIPT_NAME-$DATE.log</p><p># Colour codes<br> R=”\e[31m”<br> G=”\e[32m”<br> N=”\e[0m”<br> Y=”\e[33m”</p><p>VALIDATE() {<br> if [ $1 -ne 0 ]; then<br> echo -e “$2 $R FAILURE… $N”<br> exit 1<br> else<br> echo -e “$2 $G SUCCESS… $N”<br> fi<br> }</p><p># Migrate all repos mentioned in bitbucketReposotories.txt file<br> cat bitbucketReposotories.txt | while read REPO; do<br> # Bitbucket Repository cloning into local<br> echo -e “$Y fetching repository $REPO $N”<br> git clone — mirror <a href="https://${{secrets.BITBUCKET_USERNAME}}:${{secrets.BITBUCKET_TOKEN}}&lt;a href=">@bitbucket</a>.com/scm/${{vars.BITBUCKET_PROJECTKEY}}/$REPO.git”&gt;https://${{secrets.BITBUCKET_USERNAME}}:${{secrets.BITBUCKET_TOKEN}}<a href="http://twitter.com/bitbucket">@bitbucket</a>.com/scm/${{vars.BITBUCKET_PROJECTKEY}}/$REPO.git</p><p>VALIDATE $? “Cloning Repo $REPO”<br> echo -e “$G $REPO repository is cloned successfully and will be created on Github $N”</p><p>cd $REPO.git &amp;&gt;&gt;$LOGFILE<br> VALIDATE $? “Changed to Repo $REPO”</p><p># Push mirror to newly created repository on Github<br> REPO_TARGET=”${{vars.LOB}}-$REPO</p><p>.git”<br> echo -e “$Y $REPO repository mirroring to Github $N”</p><p>git push — mirror <a href="https://${{secrets.GITHUB_USERNAME}}:${{secrets.GITHUB_TOKEN}}&lt;a href=">@github</a>.com/$Github_ORG_NAME/$REPO_TARGET.git”&gt;https://${{secrets.GITHUB_USERNAME}}:${{secrets.GITHUB_TOKEN}}<a href="http://twitter.com/github">@github</a>.com/$Github_ORG_NAME/$REPO_TARGET.git<br> VALIDATE $? “Pushed Repo $REPO_TARGET”</p><p># Clean-up the local repository after repo migration is completed<br> cd ..<br> echo -e “$Y Deleting repository $REPO from local $N”<br> rm -rf $REPO.git &amp;&gt;&gt;$LOGFILE</p><p>echo -e “$G $REPO repository migration is successfully completed! $N”<br> done<br>```</p><p>### Explanation of the Workflow:</p><p>- **Workflow Trigger:**<br> — The workflow is triggered manually using the `workflow_dispatch` event.</p><p>- **Environment Variable:**<br> — Defines an environment variable `Github_ORG_NAME` to specify the target GitHub organization.</p><p>- **Jobs:**<br> — **Bitbucket_Github_Migration:**<br> — **Runs on:** Ubuntu latest.<br> — **Steps:**<br> 1. **Checkout Code:**<br> — Uses the `actions/checkout` action to fetch the repository code.</p><p>2. **Execute Migration Script:**<br> — A Bash script is executed inline.<br> — The script performs the Bitbucket to GitHub migration for repositories listed in `bitbucketReposotories.txt`.</p><p>## Executing the Migration:</p><p>1. Commit and push the `migrate.yml` workflow file to your GitHub repository.</p><p>2. Create a text file named `bitbucketReposotories.txt` in the root of your repository, listing the Bitbucket repositories you want to migrate.</p><p>3. Navigate to your GitHub repository, go to the “Actions” tab, and run the workflow manually by selecting “Run workflow” and triggering the `Bitbucket to GitHub Repositories Migration` workflow.</p><p>4. Monitor the workflow progress in the GitHub Actions tab. Review the logs for any errors or success messages.</p><p>5. Upon successful completion, your Bitbucket repositories will be mirrored to GitHub under the specified organization.</p><p>This GitHub Actions workflow provides an automated and efficient way to migrate multiple Bitbucket repositories to GitHub, simplifying the process and minimizing manual intervention.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3e6992331107" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>