<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by DataFairy on Medium]]></title>
        <description><![CDATA[Stories by DataFairy on Medium]]></description>
        <link>https://medium.com/@datafairy?source=rss-b354350522f5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*uv2Brt8chZioR4Hv1V43Uw@2x.jpeg</url>
            <title>Stories by DataFairy on Medium</title>
            <link>https://medium.com/@datafairy?source=rss-b354350522f5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 16:06:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@datafairy/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Getting started with data quality]]></title>
            <link>https://medium.com/towards-data-engineering/getting-started-with-data-quality-c7577ff9e6cf?source=rss-b354350522f5------2</link>
            <guid isPermaLink="false">https://medium.com/p/c7577ff9e6cf</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[DataFairy]]></dc:creator>
            <pubDate>Thu, 29 Oct 2026 19:00:59 GMT</pubDate>
            <atom:updated>2026-10-29T19:00:59.261Z</atom:updated>
            <content:encoded><![CDATA[<h4>A guide on how to get started with data quality to better understand your data and deliver more value with your data products.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/0*JS7TKNxP-nvKEWDR" /></figure><h3>What problems are quality measurements supposed to solve?</h3><p>By measuring the quality of your data products you are creating additional insights for your team. With these insights you can inform your stakeholders about the health and quality of your data products. Implemented correctly data quality checks can reveal issues in your data pipelines as soon as they appear, alert you about faulty data before it is integrated in your own systems and create realistic expectations for your stakeholders. A data quality health score of 90% could mean that the data is reliable but it might be worth to invest in further improvements to the data pipelines.</p><h3>Getting to know your data</h3><p>Before you get all excited and start defining quality checks for your data pipelines I would recommend taking a moment to really understand your data first. Without a clear idea on how your data is made up and where it comes from and what your stakeholders expect it might be difficult to come up with proper quality checks. Here is a list of things you might want to check out first.</p><h4>Schema</h4><p>The schema is the blueprint of the data table and it contains the name and data type of each column of the specific table. In un-structured (NoSQL) tables this might not be as simple but for now we will work with the traditional structured tables (SQL).</p><p><strong>Creating a snowflake table using SQL and defining the schema:</strong></p><pre>CREATE OR REPLACE TABLE MENU<br>(<br>menu_id NUMBER(19,0),<br>menu_type_id NUMBER(38,0),<br>menu_type VARCHAR(16777216),<br>menu_item_id NUMBER(38,0),<br>menu_item_name VARCHAR(16777216),<br>item_category VARCHAR(16777216),<br>item_subcategory VARCHAR(16777216),<br>cost_of_goods_usd NUMBER(38,4),<br>sale_price_usd NUMBER(38,4)<br>);</pre><p>Defining a strict schema where the data types reflect the data as precise as possible (not using the max possible size for integers for example) can restrict the data that is inserted into a table. Using the expression “not Null” can also ensure that the columns are filled and empty rows are rejected.</p><p>Forcing all ID’s to be filled using NOT NULL:</p><pre>CREATE OR REPLACE TABLE MENU<br>(<br>menu_id NUMBER(19,0) NOT NULL,<br>menu_type_id NUMBER(38,0) NOT NULL,<br>menu_type VARCHAR(16777216),<br>menu_item_id NUMBER(38,0) NOT NULL,<br>menu_item_name VARCHAR(16777216),<br>item_category VARCHAR(16777216),<br>item_subcategory VARCHAR(16777216),<br>cost_of_goods_usd NUMBER(38,4),<br>sale_price_usd NUMBER(38,4)<br>);</pre><p>When I look at a table schema I want to get a first expression of the data I will be working with. This can be achieved by creating a schema that represents the data as closely as possible using restrictions on data types, comments and constraints.</p><p><strong>Adding comments and constraint to better refelect the actual data:</strong></p><pre>CREATE OR REPLACE TABLE MENU<br>(<br>menu_id NUMBER(19,0) NOT NULL,<br>menu_type_id NUMBER(1,0) NOT NULL, -- this can either be 1, 2 or 3<br>menu_type VARCHAR(1), -- this can either be A, B or C<br>menu_item_id NUMBER(19,0) NOT NULL,<br>menu_item_name VARCHAR(50), -- menu-times can have &quot;longer&quot; names <br>item_category VARCHAR(10), -- categories are single names<br>item_subcategory VARCHAR(10),  -- sub-categories are single names<br>cost_of_goods_usd NUMBER(10,2), -- cost of goods can be large but not unlimited<br>sale_price_usd NUMBER(3,2) -- prices should not exceed 999.99$<br>)<br><br>CONSTRAINT check_type_id CHECK (menu_type_id in [A, B, C])<br>CONSTRAINT check_type CHECK (menu_type in [A, B, C])<br>CONSTRAINT chk_positive_price CHECK (sale_price_usd &gt; 0)<br>CONSTRAINT chk_positive_cost CHECK (cost_of_goods_usd &gt; 0);<br><br></pre><h4>Documentation</h4><p>While the schema describes a table in a concise way, documentation does the opposite. In your data documentation you want to include every information necessary to understand your data products. Where did the data come from, what do we expect from the data, who will be using the data and who is the owner of the data product?</p><p>Add information to your documentation that you can’t add to your schema definition.</p><p>For more info on this topic: <a href="https://www.thedataops.org/data-documentation/">https://www.thedataops.org/data-documentation/</a>.</p><h4>Lineage</h4><p>Once you have a grip on what to expect from your data you want to look at the lineage. It can be especially interesting to compare data schemas. Just based on that you can find inconsistencies that need to be fixed, see where mapping happened (changes in data types from one table to the other) and see where multiples tables have been joined into a new column.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/436/1*AEsCYPjcdDSHiwDbs7UIlQ.png" /><figcaption>Snowflake lineage chart</figcaption></figure><p>Without looking at the data yet you could already have a good idea what to expect.</p><h4>The data itself</h4><p>Finally it’s time to look at the data itself. Compare the data to your schema, run some queries to find the unique values and define constraints that can be used in your quality measurements. Write down questions for your domain experts about the specific restrictions and expectations for your table you need to double check.</p><p><strong>Some simple queries to analyse tables in SQL:</strong></p><pre>SELECT DISTINCT(column_1) FROM table_name;<br>SELECT MIN(column1), MAX(column1) FROM table_name;<br>SELECT column1 FROM table_name WHERE condition;</pre><p>Once you have checked all the boxes you should have a better grip on your data and be ready to define data quality measurements. Check out the second part of this post, and until then happy coding!</p><h4>Summary</h4><p>Data quality can be an intimidating topic to tackle and feel like an extra amount of work with little benefits. With this article I wanted to get you started with data quality by first understanding your data before defining quality measurement configurations for all your tables. I do believe this understanding will make you appreciate the value of quality measurements more and help you create them faster and better.</p><blockquote>If you found this article useful, clap, comment and please follow me</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c7577ff9e6cf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-data-engineering/getting-started-with-data-quality-c7577ff9e6cf">Getting started with data quality</a> was originally published in <a href="https://medium.com/towards-data-engineering">Towards Data Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Switching to secure and encrypted email — Proton Mail & Pass]]></title>
            <link>https://medium.com/@datafairy/switching-to-secure-and-encrypted-email-proton-mail-pass-3cb299a310ee?source=rss-b354350522f5------2</link>
            <guid isPermaLink="false">https://medium.com/p/3cb299a310ee</guid>
            <category><![CDATA[cybersecruity]]></category>
            <category><![CDATA[protonmail]]></category>
            <category><![CDATA[proton-pass]]></category>
            <dc:creator><![CDATA[DataFairy]]></dc:creator>
            <pubDate>Thu, 03 Apr 2025 12:35:53 GMT</pubDate>
            <atom:updated>2025-04-04T08:29:59.048Z</atom:updated>
            <content:encoded><![CDATA[<h3>Switching to secure and encrypted email — Proton Mail &amp; Pass</h3><h4>What I learned switching from gmail to Proton Mail. A journey to email account security.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1002/1*bmPu6ka1408Z7fJ3J07tyg.png" /></figure><h3>Introduction</h3><p>As you may have seen in my last article <a href="https://medium.com/@datafairy/looking-at-the-global-tech-market-from-a-european-perspective-f66f7b805dbe">Looking at the global tech market from a European perspective | by DataFairy | Mar, 2025 | Medium</a>, I do not believe we should rely that heavily on U.S. tech industries. Their market dominance is unproportional to the U.S. population and their security standards are iffy at best.</p><p>You might have also realised that the Trump administration with Elon Musk as “president” aren’t going anywhere soon. Strenghtening the European tech industry isn’t only a good idea but a necessity for the future.</p><p>I decided to start my tech transition with one of the worst data collecting machines: Google. The last month I switched from Google search, Google Chrome and Gmail to Ecosia, DuckDuckGo and Proton. In this article I want to share with you how that went and what I learned.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*f3jKx8k9xsgXEWUBWADqwg.png" /><figcaption>Ecoasia web browser</figcaption></figure><p>Switching browsers and search engines isn’t as big a deal as switching email accounts. For the sake of keeping you guys entertained I will go directly to the email part.</p><h3>Inventarisation</h3><p>The first thing I had to do is to take stock. How many accounts have a I created, on how many mailing lists am I and what is still valid and useful.</p><p>When I created my Proton account I also switched to Proton Pass. Before that I had my passwords stored in my browser and in KeyPass. Once I exported all of them and imported them to Proton Pass I had more than 120 entries. This felt quite overwhelming and those were just username and password combinations. I was also getting emails from mailing lists.</p><p>I felt tired even before I started.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/585/1*J-HveBe4nn-KMh4aabobEw@2x.jpeg" /><figcaption>After cleaning I have 91 accounts left</figcaption></figure><h3>Proton Mail</h3><p>Next I had to figure out how to get started with Proton Mail. Proton Mail makes it really easy to forward your emails from other providers such as Gmail. You can also move some of the history over to Proton while setting up the forwarding. In a few minutes I was getting mails in Gmail and Proton at the same time. Now I could get started on reducing the mails coming to Gmail.</p><h4>Aliases</h4><blockquote><strong>ProtonMail allows you to create email aliases</strong>:</blockquote><blockquote>Use a plus sign (+) in your username to create aliases. For example, if your address is alicejones@proton.me, you could create aliases like alicejones+shop@proton.me or alicejones+news@proton.me.</blockquote><blockquote>Paid ProtonMail users can create unlimited aliases, while free account users get up to 10 of them.</blockquote><blockquote>Aliases are randomly-generated email addresses that forward emails to your main inbox, protecting your actual email address.</blockquote><p>First I wanted to create aliases, which are email addresses that redirect your mail to your Proton account. Let’s say my email is xxx@proton.me but I don’t want to tell you that cause you are an annoying recruiter who will stalk me. I create an alias via Proton mail that is something like goaway+stalker@passinbox.com. Now I give this alias out and keep my real mail private.</p><p>The first action on Proton Mail is to create aliases for different use cases such as mailinglist, shopping, login, banking, social networks and so forth.</p><p>This was the easy part. Next it was time to go through all the accounts I have made and switch them over to one of the aliases. 120 + accounts, that wasn’t gonna happen in one day.</p><h4>Weak passwords</h4><p>The benefit of this inventarisation is that I got to go through all my accounts and check the strength of my passwords. Proton Pass tells you very clearly if your password is strong enough and makes it easy to generate a new one. After going through all my accounts (10–20 a day) I not only have a distinct email+password combination for every account now, but all my passwords are very strong now.</p><h3>Going down the rabbit hole</h3><p>All accounts? Well almost all accounts. With every good intention there are always roadblocks. Some small, some big and some a bit smelly.</p><h4>Smelly</h4><p>One thing I learned is that streaming services do not like us if we don’t pay a subscription. They do not only dislike us but they also block access to our account information. When I tried to change my email address for Disney+, Hulu, Netflix and HBOmax I couldn’t. I didn’t have an active sub and I wasn’t gonna buy one. But without a sub you cannot get beyond the “choose a sub” screen on their websites. You cannot see your account information nor change your password.</p><p>I don’t feel like streaming any shows right now.</p><h4>Create your account once …</h4><p>seems to be the mentality of some companies. They don’t seem to consider people wanting to change anything except for addresses or phone numbers. The field for changing your email address is permanently greyed out and you have to contact customer support and hope they will help you.</p><p>In these cases I really would like to have a <strong>delete account </strong>button.</p><blockquote>Unfortunately, I can’t help you any further. I can’t change the e-mail addresses if I don’t have an order for this. — CEO XXX</blockquote><p>At the end I deleted a lot of useless accounts like Instagram, Twitter (X) and some webshop accounts.</p><p>If you want to test your patience try deleting (disabeling) your X account, great fun.</p><h3>Summary</h3><p>After switching to an encrypted email and password service I have a much better grip on all of my accounts. I got to see which services make it easy for us customers to be in control of our information and which do not. Getting rid of useless accounts and strengthening my passwords has improved my online security. Next time I have to make an account I will think twice about giving away personal information.</p><blockquote>If you found this article useful, clap, comment and please follow me</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3cb299a310ee" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Looking at the global tech market from a European perspective]]></title>
            <link>https://medium.com/@datafairy/looking-at-the-global-tech-market-from-a-european-perspective-f66f7b805dbe?source=rss-b354350522f5------2</link>
            <guid isPermaLink="false">https://medium.com/p/f66f7b805dbe</guid>
            <category><![CDATA[it]]></category>
            <category><![CDATA[mobile-apps]]></category>
            <category><![CDATA[europe]]></category>
            <category><![CDATA[tech-industry]]></category>
            <category><![CDATA[global-market]]></category>
            <dc:creator><![CDATA[DataFairy]]></dc:creator>
            <pubDate>Thu, 06 Mar 2025 11:57:12 GMT</pubDate>
            <atom:updated>2025-04-04T08:29:29.049Z</atom:updated>
            <content:encoded><![CDATA[<h4>Looking at the U.S. global dominance in the tech industry and how we contribute to this inbalance from a European perspective.</h4><p>So let’s ask ourselves first, who controls the tech world?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MzFPcU98dii-so7sYoad1A.png" /><figcaption>Tech companies ranked by market cap worldwide. <a href="http://companiesmarketcap.com/tech/largest-tech-companies-by-market-cap/">http://companiesmarketcap.com/tech/largest-tech-companies-by-market-cap/</a></figcaption></figure><p>The tech market is worth about 32.861 trillion dollars. The first nine U.S. companies in the above list make up 55% of this market. The first European company on this list is the German SAP with 1.034% of the market cap. Looking at this numbers I find this quite shocking and worrisome. The global economy is completely dependent on the USA and its suppliers. If you look at the global market overall, not just tech, you will find that the top ten spots on the list are dominated by U.S. tech giants.</p><p>On March 2nd 2025 the U.S. population counted around <strong>346.659.484 </strong>people which is <strong>4.22%</strong> of the global population.<strong> </strong>Not a small number but also not as huge as China (<strong>1.417.162.818) </strong>or India (<strong>1.459.541.049) </strong>with nearly 1.5 billion people each.</p><p>Europe and the UK have a population of <strong>744.625.701 + 69.414632 = 814.040.333 </strong>people, which is about <strong>9.21%</strong> of the worlds population. That is almost<strong> 2.35 times the U.S. population</strong>.</p><p><a href="https://www.worldometers.info/">https://www.worldometers.info/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/656/1*pSRmKpMGBt2GqIae5ZZ8WQ.png" /><figcaption><a href="https://www.visualcapitalist.com/the-109-trillion-global-stock-market-in-one-chart/">https://www.visualcapitalist.com/the-109-trillion-global-stock-market-in-one-chart/</a></figcaption></figure><p>Looking at the global stock market the numbers speak for themselves. The USA is very dominant with more than 42% and other countries struggle to keep up or get even close.</p><p>The USA is a democratic country with a long history of alliances around the world. A strong system of check and balances has ensured that not too much power is given to one single institution or person. For the longest time the West has felt very save being an important ally to the USA.</p><p>With the rise of the billionaire class, the concentration of wealth in the hand of the very very few and the political turmoil the West finds itself in these days I do ask myself if we can safely rely on this division of power in favor of the USA. I am not a politician nor do I intend to make politics on this site, but as a European citizen I do ask myself if we have been too comfortable on this side of the Atlantic.</p><p>Our entire technological stack is being provided by companies outside of Europe. If for some reason one day the global conditions would change and we couldn’t access these technologies anymore, there is not much we can fall back on. At least not in a short amount of time.</p><p>In this article I want to explore the alternatives, specifically the tech alternatives we could implement into our daily lives that run in Europe and have been created by European companies.</p><h3>Diving into my own tech stack:</h3><p>This is a list of apps and tools I encounter daily:</p><ul><li>Microsoft Windows</li><li>IPhone OS from Apple</li><li>Gmail</li><li>Microsoft Edge</li><li>Google Chrome</li><li>Google Search</li><li>Intel/AMD/NVIDIA processors</li></ul><p>These are the software and hardware tools that define how I interact with the internet.</p><p>Now let’s look at a list of Apps I have and use on a daily basis:</p><ul><li>Medium</li><li>MS Teams, Slack</li><li>Authenticator</li><li>Banking and payment apps</li><li>Travel apps</li><li>Shopping and food delivery apps</li><li>Social apps like Linkedin and Whatsapp</li><li>Governement apps</li><li>Home apps (hue, dyson, Bose connect)</li><li>Spotify, Apple TV, YouTube</li><li>Productivity apps: Sweepy, Timo, Toggl Track</li><li>Google Maps</li></ul><p>Even though I am not using Facebook, Instagram, Amazon or Tick Tock I do find that I very often use the default apps like Google maps. The alternatives out there exists but are far less popular or rank lower on the “App” stores. I checked out this website on european alternatives and deleted some of my “standard” apps for more local choices.</p><p><a href="https://european-alternatives.eu/">https://european-alternatives.eu</a></p><p>I was positively surprised to find that in Europe we have enough alternatives to all the important tools I use every day. There is of course a cost to switching apps and technologies. Apps for social media and communication require more than one user to be of any use. Tech giants like Microsoft that dominate not only everyone’s work environment but are also part of an entire tech ecosystem are much more difficult to replace short term.</p><p>Europe has the tools and know how “in-house”. If we want to we can slowly but successfully move away from our dependent position towards being more self-reliant and having a strong IT landscape. This would create jobs, increase security and strengthen the European economy.</p><p>Think about it.</p><blockquote>If you found this article useful, clap, comment and please follow me</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f66f7b805dbe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Handovers — The blind spot of IT and data teams]]></title>
            <link>https://medium.com/@datafairy/handovers-the-blind-spot-of-it-and-data-teams-98b29c396d37?source=rss-b354350522f5------2</link>
            <guid isPermaLink="false">https://medium.com/p/98b29c396d37</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[project-management]]></category>
            <category><![CDATA[handover]]></category>
            <category><![CDATA[technical-leadership]]></category>
            <category><![CDATA[data-project]]></category>
            <dc:creator><![CDATA[DataFairy]]></dc:creator>
            <pubDate>Tue, 04 Feb 2025 14:01:28 GMT</pubDate>
            <atom:updated>2025-04-07T19:08:40.983Z</atom:updated>
            <content:encoded><![CDATA[<h3>Handovers — The blind spot of IT and data teams</h3><h4>How to plan and execute handovers effectively to never loose knowledge in your team</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/1*918dLomTMZtqdX9F1smDqg.png" /><figcaption>A developer on the run</figcaption></figure><h3>Handovers — What is the issue?</h3><p>As a data, software and devops engineer I have seen the following scenario happening in every team at every company all the time: There is this one person who holds all the knowledge of a product or solution, doesn’t write enough documentation and always has an excuse not to share this knowledge (we are all busy). And when this person leaves the company everyone suddenly runs around in a panic to get the information they couldn’t access until now, documented and shared.</p><p>Another fun example: when you join a new team and you start looking at their code base. You check the documentation and you will find that the author has not updated it for months or years and/or is not working at the company anymore. Your team members will struggle to help you because they never did a proper handover. Excuses for bad documentation run rampant in the IT and data industry.</p><p>If you recognise these issues and you don’t know how to fix them you need to look at the way your team handles handovers.</p><h4>Disclaimer</h4><p>In the following I will refer to the developers of the product/solution as the <strong>development team</strong>. Those could be external consultant or internal colleagues. The companies engineers or data professional will be refered to as the <strong>in-house team</strong> that receives the product/solution and has to keep it alive and running.</p><h4>Why teams fail at handovers?</h4><p>Teams fail at handovers for several reasons but foremost because of these two:</p><ul><li>A missing process that works for them</li><li>Priorities or a lack of</li></ul><p>The main reason, in my experience, why handovers fail is that teams don’t have a process for them. Handovers are semi-well done and often the handover starts when a project is finished. This is much too late in a lot of cases. In my experience a succesful handover starts at the beginning of the project.</p><p>Handovers are very often forgotten about or are done in the last week before a development team member leaves the team/company. The priority then is very high but because of the time pressure and bad planning everyone involved needs to suddenly make time in their busy schedules. Documentation might be written but is there ever enough time to review it properly? Handovers need to be planned ahead of time and prioritised properly for the entire team to work well.</p><h4>What are handover processes for?</h4><p>A handover process is not only about writing documentation but also to:</p><ul><li>Ensure that the knowledge stays within the in-house team when someone leaves</li><li>Ensure members of the development team share project critical knowledge with each other and the in-house team</li><li>Ensure products and solutions are finished and reviewed</li><li>Ensure the in-house team can support the products after the handover</li><li>Ensure documentation is written and updated where necessary</li><li>Ensure there is enough time for the handover between the product delivery and the development team member leaving</li></ul><h3>Handovers — done better</h3><p>Here are some guideline to do handovers better and more effectively so that your (in-house) team can share knowledge and keep products running successfully.</p><h4>Plan in advance</h4><p>One secret to succesful handovers is to start them early. By this I do not mean that we should handover projects that aren’t finished yet. Rather we should define what we expect at a handover (documentation, presentations, reviews) and communicate this with the executing party (teams, consultants, project managers, product owners) at the start of the project. The managers (PM, PO) need to realise the importance of handovers so that they can plan extra time in advance. The handover can then be done properly and get priority.</p><p>Planning in advance with a list of objectives will create clear expectations that will help guide the development team so that they can create a succesful product that can be maintained long-term.</p><h4>Key principles</h4><ul><li>Clearly communicate your expectations in advance</li><li>Leave room for the in-house team to review a solution and ask questions</li><li>Make sure the solutions has run succesfully for a few weeks before it is declared finished</li><li>Involve in-house team members early in the development process by letting them review pull requests for the code base</li><li>Let the development team give presentations a long the way to monitor progress and have in-house team members attend to ask questions and give feedback</li><li>Never rely on only one in-house team member to be the keeper of all the knowledge, always share as much knowledge as possible with the entire in-house team</li></ul><h3>Final remarks</h3><p>For years I worked in a department that run short data science projects (up to one year) with a lot of consultants as their main developers. One of the first things I set up was a definition for handovers. By communicating our expectations early we could make sure that the projects could be reused by a new team coming in. All domain specific knowledge stayed within our company.</p><p>This saved us a lot of time and headaches. New teams could start out by reading documentation and we knew that the code base was tested and reviewed. The new developers could just start off where others had left.</p><p>Without a clear process for handovers we would have definitely wasted a lot of time at the beginning of each project and some projects might have never been continued.</p><p>I currently work in a team that creates long-term data solutions. The challenge is the same, how to share knowledge and make sure we can run our solutions succesfully for a long time. A solid handover process is again a very good tool to ensure this.</p><p>It doesn’t matter if you continuesly run short projects or need to keep solutions running in production. Handovers are one of the key tools to long-term success in IT and data teams.</p><blockquote>If you found this article useful, clap, comment and please follow me</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=98b29c396d37" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Experimenting with Python threads]]></title>
            <link>https://medium.com/@datafairy/experimenting-with-python-threads-88f66b002acf?source=rss-b354350522f5------2</link>
            <guid isPermaLink="false">https://medium.com/p/88f66b002acf</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[threading-in-python]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[DataFairy]]></dc:creator>
            <pubDate>Mon, 08 Jul 2024 13:49:44 GMT</pubDate>
            <atom:updated>2024-07-11T08:35:42.502Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to thread and how not to thread in Python</h4><h3>Introduction</h3><p>Understanding threads in Python is crucial when processing with outputs from events such as I/O tasks. Using threads allows us to write code that potentially runs faster than without the threading library. When working with threads it is important to understand that threads in Python work differently from threads in Java for instance due to the limitation enforced by the GIL. This requires Python developers to <strong><em>thread</em></strong> carefully to avoid common pitfalls.</p><p><strong>This is how a thread executes as a function of time:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*mIPNEXCasZQlifynI3t2Zg.png" /><figcaption><a href="https://www.pythontutorial.net/python-concurrency/python-threading/">A Practical Guide to Python Threading By Examples (pythontutorial.net)</a></figcaption></figure><h3>What are threads in Python?</h3><p>A <strong>thread</strong> is a separate flow of execution within a program that can run concurrently with other threads, allowing multiple operations to (appear to) occur simultaneously.</p><pre>import threading<br><br>def hello_world():<br>    print(&quot;Hello, world!&quot;)<br><br>t = threading.Thread(target=hello_world)<br>t.start()</pre><h3>Multithreading in Python</h3><blockquote>Python virtual machine is not a thread-safe interpreter, meaning that the interpreter can execute only one thread at any given moment. This limitation is enforced by the Python Global Interpreter Lock (GIL), which essentially limits one Python thread to run at a time. In other words, GIL ensures that only one thread runs within the same process at the same time on a single processor.</blockquote><blockquote>Basically, threading may not speed up all tasks.</blockquote><pre>import threading<br>import time<br><br># Function to be executed in the thread<br>def print_numbers():<br>    for i in range(5):<br>        time.sleep(1)  # Simulating some work<br>        print(f&quot;Thread 1: {i}&quot;)<br><br>def print_letters():<br>    for letter in &quot;abcde&quot;:<br>        time.sleep(1)  # Simulating some work<br>        print(f&quot;Thread 2: {letter}&quot;)<br><br># Create two threads<br>t1 = threading.Thread(target=print_numbers)<br>t2 = threading.Thread(target=print_letters)<br><br># Start the threads<br>t1.start()<br>t2.start()<br><br># Wait for both threads to finish<br>t1.join()<br>t2.join()<br><br>print(&quot;Done!&quot;)</pre><pre>Thread 1: 0<br>Thread 2: a<br>Thread 2: b<br>Thread 1: 1<br>Thread 2: c<br>Thread 1: 2<br>Thread 1: 3<br>Thread 2: d<br>Thread 1: 4<br>Thread 2: e<br>Done!</pre><h3>When should threads be used?</h3><p><strong>Good candidates:</strong> Tasks that spend much of their time waiting for external events such as I/O bound tasks.</p><p><strong>Bad candidates:</strong> Problems that require heavy CPU computation and spend little time waiting for external events.</p><h4>Pitfalls</h4><p><strong>only one thread in Python:</strong></p><p>As mentioned above only one thread can execute simultaneously in Python. To make multiple threads work some smart coding is required:</p><pre># Using lists<br>threads = list()<br>for index in range(3):<br>    logging.info(&quot;Main    : create and start thread %d.&quot;, index)<br>    x = threading.Thread(target=thread_function, args=(index,))<br>    threads.append(x)<br>    x.start()<br><br>for index, thread in enumerate(threads):<br>    logging.info(&quot;Main    : before joining thread %d.&quot;, index)<br>    thread.join()<br>    logging.info(&quot;Main    : thread %d done&quot;, index)<br><br># Using ThreadPoolExecutor<br>with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:<br>    executor.map(thread_function, range(3))</pre><p><strong>multiple background threads cause exceptions:</strong></p><p>Providing the argument daemon=True to a thread will flag it as a background thread. This thread will terminate when the VM/container it is running on terminates. The only issue with this is that if more than one thread is flagged as background thread the entire program will not work.</p><p><strong><em>This is code that runs two threads in the foreground:</em></strong></p><pre>import threading<br><br><br>def countSeconds():<br>    print(&quot;Second&quot;)<br>    threading.Timer(1, countSeconds).start()<br><br>def countTenSeconds():<br>    print(&quot;Ten seconds passed&quot;)<br>    threading.Timer(10, countTenSeconds).start()<br><br><br>threading.Timer(1, countSeconds).start()<br>threading.Timer(10, countTenSeconds).start()</pre><p>The output is something like this:</p><pre>Ten seconds passed<br>Second<br>Second<br>Second<br>Second<br>Second<br>Second<br>Second<br>Second<br>Second<br>Second</pre><p>When running the first function with daemon=True</p><pre>def countSeconds():<br>    print(&quot;Second&quot;)<br>    threading.Timer(1, countSeconds, daemon=True).start()</pre><p>The result is the following:</p><pre>Ten seconds passed<br>Ten seconds passed</pre><p>Adding <em>daemon=True</em> to both functions result in multiple exceptions:</p><pre>TypeError: Timer.__init__() got an unexpected keyword argument &#39;daemon&#39;<br>Ten seconds passed<br>Exception in thread Thread-2:<br>Traceback (most recent call last):<br>  File &quot;/usr/lib/python3.10/threading.py&quot;, line 1016, in _bootstrap_inner<br>    self.run()<br>  File &quot;/usr/lib/python3.10/threading.py&quot;, line 1378, in run<br>    self.function(*self.args, **self.kwargs)<br>  File &quot;/mnt/c/python_threading_plg/src/count.py&quot;, line 10, in countTenSeconds<br>    threading.Timer(10, countTenSeconds, daemon=True).start()<br>TypeError: Timer.__init__() got an unexpected keyword argument &#39;daemon&#39;</pre><p>This is something to keep in mind when combining multiple threads in Python.</p><p><strong>race conditions:</strong></p><p>When two or more threads are trying to access a shared resource we talk about race conditions. An example would be two threads writing to a database simultaneously.</p><p><strong>deadlocks:</strong></p><p>When working with locks in threads we can acquire a lock and release a lock. We cannot acquire a lock that has not been released first or the resource has to be unlocked.</p><p>When two acquire operations are run without a release operation we end up in a deadlock.</p><h4>Interesting facts on the threading library in Python:</h4><ol><li>threading.Lock synchronizes threads and allows only one thread at a time to access a resource.</li><li>The Python threading library takes advantage of only <strong>one CPU</strong> (or core) simultaneously.</li><li>The Thread.join() method does wait for the thread to finish.</li><li>Race conditions are two threads incorrectly accessing a shared resource.</li><li>Using ThreadPoolExecutor as a context manager is a good idea cause it manages the start up of all threads and waits for them all to complete.</li><li>Once you start a threading.Timer, you can stop it before it expires.</li></ol><h4>threading vs multiprocessing</h4><p>Threading is a form of concurency that happens on a single processor. Multiprocessing in contrast can make use of all the available cores.</p><p>Multiprocessing becomes interesting when you have to manage I/O bound problems or CPU-heavy workloads. The second is self-explanatory while the first deals with slow network connections or file systems.</p><p>To solve the I/O problem we would be overlapping the times spent waiting for the i/o devices. For the CPU problem we would have to find ways to do more computations in the same amount of time.</p><h4>resources</h4><p><strong>Code examples:</strong></p><ul><li><a href="https://github.com/datafairy-azure/python_threading_plg">datafairy-azure/python_threading_plg (github.com)</a></li></ul><p><strong>References:</strong></p><ol><li><a href="https://realpython.com/intro-to-python-threading/">An Intro to Threading in Python — Real Python</a></li><li><a href="https://realpython.com/python-concurrency/">Speed Up Your Python Program With Concurrency — Real Python</a></li><li><a href="https://www.dataquest.io/blog/multithreading-in-python/">Multithreading in Python: The Ultimate Guide (with Coding Examples) (dataquest.io)</a></li></ol><blockquote>If you found this article useful, clap, comment and please follow me.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=88f66b002acf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Container image security: How to find vulnerabilities and fix them]]></title>
            <link>https://medium.com/@datafairy/container-image-security-how-to-find-vulnerabilities-and-fix-them-02da66ac2023?source=rss-b354350522f5------2</link>
            <guid isPermaLink="false">https://medium.com/p/02da66ac2023</guid>
            <category><![CDATA[containerization]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[trivy]]></category>
            <dc:creator><![CDATA[DataFairy]]></dc:creator>
            <pubDate>Thu, 27 Jun 2024 13:29:36 GMT</pubDate>
            <atom:updated>2025-02-03T21:25:01.721Z</atom:updated>
            <content:encoded><![CDATA[<h4>Vulnerabilities in container images are not only hidden security risks but can also pose a challenge when you want to fix them. In this article we dive into the methods of discovering and fixing these security risks.</h4><h3>Introduction</h3><p>As a data engineer applications based on docker images and containers have always been part of my work. It is quite difficult to imagine microservices without them. Serverless containers are on the rise according to <a href="https://www.datadoghq.com/container-report/">datadoghq.com</a> and Kubernetes is still growing in popularity: <a href="https://www.datadoghq.com/container-report-2022/">9 Insights on Real-World Container Use | Datadog (datadoghq.com)</a>.</p><p>We can safely assume that containers are here to stay and it is important to consider the security impacts they bring to your infrastructure. In this article I want to dive into security scanning of container images and the tools to do it.</p><h4><strong>Docker image security scanning</strong></h4><blockquote><strong>Docker image security scanning</strong> is a process of identifying known security vulnerabilities in the packages listed in your Docker image. This gives you the opportunity to find vulnerabilities in container images and fix them before pushing the image to Docker Hub or any other registry.</blockquote><blockquote>by <a href="https://snyk.io/learn/docker-security-scanning/">Docker Security Scanning Guide 2023 | Snyk</a></blockquote><p>The idea here is to find vulnerabilities in container images using the available tools, fix those security issues and push the updated images to the container registry of your choice.</p><h4>Classifications</h4><p>Vulnerabilities are classified by severity and impact. The catergories are Critical, High, Medium and Low. According to MS Edge Copilot the defintions are as follows:</p><p><strong>Critical</strong></p><ul><li>Critical vulnerabilities pose the highest risk.</li><li>Exploiting them can lead to severe consequences, such as unauthorized access, data breaches, or system compromise.</li><li>Immediate attention and mitigation are essential.</li></ul><p><strong>High</strong></p><ul><li>High-severity vulnerabilities are significant but not as critical as critical ones.</li><li>They may allow attackers to compromise security or gain unauthorized access.</li><li>Prompt action is still necessary.</li></ul><p><strong>Medium</strong></p><ul><li>Medium-severity vulnerabilities have a moderate impact.</li><li>They might expose sensitive information or allow limited unauthorized access.</li><li>Address them promptly to reduce risk.</li></ul><p><strong>Low</strong></p><ul><li>Low-severity vulnerabilities have minimal impact.</li><li>They may not directly compromise security but should still be addressed to maintain a robust system.</li></ul><h4>Example of vulnerabilities in container images</h4><p>Let’s look at an example using the python:3.10.12-slim docker image. This is not the latest version of python:3.10-slim and we can expect some vulnerabilities that have been fixed in newer versions.</p><p><strong>To check for vulnerabilities I used docker desktop:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O82g5uJhCZ3kq_fFerPHNw.png" /></figure><p><strong>This image consists of 2 images:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/773/1*HW5o5hhGN6jVj7nZgOhw9Q.png" /></figure><p>We can see the vulnerabilities for each image on the right side and in the colors grey to red depending on their severity.</p><p><strong>Clicking the vulnerability itself we can get more information on it:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/1*LWvR_-dfrDg5rgf9rf4T0w.png" /></figure><p>More on the <a href="https://www.first.org/cvss/">CVSS score</a> here:</p><blockquote>The Common Vulnerability Scoring System (CVSS) provides a way to capture the principal characteristics of a vulnerability and produce a numerical score reflecting its severity. The numerical score can then be translated into a qualitative representation (such as low, medium, high, and critical) to help organizations properly assess and prioritize their vulnerability management processes.</blockquote><h4>How to find vulnerabilities</h4><p>There are different tools for finding vulnerabilities, here some open-source examples:</p><ul><li>docker desktop</li><li>trivy</li><li>trivy filtering <a href="https://aquasecurity.github.io/trivy/v0.52/docs/configuration/filtering/">Filtering — Trivy (aquasecurity.github.io)</a></li><li>trivy reports</li></ul><h4>Fixing images pulled from a public registry</h4><p>There are different ways to fixing vulnerabilities:</p><ul><li>You can get an all-in-one paid solution just like Snyk.</li><li>You can do it manually by upgrading your docker image to the latest version that contains less vulnerabilities.</li><li>You can create a custom image where the problematic dependencies have been removed or upgraded (which you can do manually or with a tool like copa).</li></ul><p><a href="https://project-copacetic.github.io/copacetic/website/">Introduction | Copacetic (project-copacetic.github.io)</a></p><h4>Fixing images</h4><p><strong>Example python:3.10.12-slim</strong></p><p>This image contains 39 vulnerabilities of which 2 are critical. Here we want to fix these 2.</p><p>These are the commands I have used in the order I have executed them:</p><pre># Checking for vulnerabilities that are critical<br>trivy image python:3.10.12-slim --severity CRITICAL</pre><pre># Creatingthe buildx environment<br>docker buildx create --name demo</pre><pre># Running the patching of the image<br>copa patch -r python:3.10.12-slim.json -i docker.io/library/python:3.10.12-slim -t 3.10.12-slim-patched --addr buildx://demo</pre><pre># Checking for vulnerabilities<br>trivy image --vuln-type os --ignore-unfixed python:3.10.12-slim-patched</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gEotMOrQhgjHEUkN8layAw.png" /></figure><p>As a result we get a new image where the fixable vulnerabilities have been removed. All is left is to upgrade the pip version.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*Hh1Y-xaQ_h7QldN3DYp_Pg.png" /></figure><p>One way to do this is to add a requirements.txt file with pip&gt;=24.0 and add it to the dockerfile when using the python:3.10.12-slim image:</p><pre>COPY requirements.txt /root/requirements.txt</pre><p>To replicate my results you will have to install trivy and copa. Follow the instructions on their websites.</p><h3>Summary</h3><p>In this article we have learnd about docker security scanning. We have looked at container image vulnerabilities and it’s classifications. We used docker desktop and trivy to scan a container image for vulnerabilities. Finally we used copa to create custom image that has fixed the vulnerabilities.</p><blockquote>If you found this article useful, clap, comment and please follow me.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=02da66ac2023" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Codium AI — Writing Python tests with an AI assistant — Part 1]]></title>
            <link>https://towardsdev.com/codium-ai-writing-python-tests-with-an-ai-assistant-part-1-ba68c1234c97?source=rss-b354350522f5------2</link>
            <guid isPermaLink="false">https://medium.com/p/ba68c1234c97</guid>
            <category><![CDATA[phind]]></category>
            <category><![CDATA[test-driven-development]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[codiumai]]></category>
            <dc:creator><![CDATA[DataFairy]]></dc:creator>
            <pubDate>Sat, 13 Jan 2024 10:20:34 GMT</pubDate>
            <atom:updated>2024-01-15T08:04:47.450Z</atom:updated>
            <content:encoded><![CDATA[<h3>Codium AI — Writing Python tests with an AI assistant — Part 1</h3><h4>CodiumAI is a tool to help you write Python tests by generating them with the click of a button. In this article I walk you through my experience using the tool in VSCode.</h4><h3>Intro Codium AI</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/227/1*wTcc6Z6wPQETJsUDuXs9lQ.png" /></figure><blockquote>At <a href="https://www.codium.ai/about/#our-mission">CodiumAI</a>, we’re creating a new code integrity paradigm with the power of artificial and human intelligence. Our vision is to enable developers to build faster with zero bugs.</blockquote><blockquote>CodiumAI’s first tool is an IDE extension that interacts with the developer to generate meaningful tests and code explanations for busy devs.</blockquote><p>So this is how it looks like in VSCode when I use the tool for creating tests:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/1*i3hOqWFgKrDTLRyf1R7L6w.png" /><figcaption>A function that reads in a yaml file.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S2WnIWfkTdqIMtwZXvsY3Q.png" /><figcaption>The tests open up in VSCode and can be run and saved to file.</figcaption></figure><h3>Getting started</h3><p>Before writing any tests I want to define the scope of my experiment. I will be testing the azure-mlops code I had written for the MLOps pipeline I described in previous posts.</p><p><a href="https://medium.com/@datafairy/creating-a-mlops-pipeline-with-azure-machine-learning-service-b960fb80a7d1">Creating a MLOPS pipeline with Azure Machine Learning Service</a></p><p>The code contains a package called mlops which has four Python files for each step of the MLOps process and one utils file with helper functions for AzureML.</p><p>To get an idea of how good I am testing my code I will then setup coverage. Afterwards I want to start with a simple test case and work my way up to more difficult examples. As reference I will use the tests given in Microsofts github repo with examples for AzureML.</p><h4>Setting up coverage</h4><p>First let’s install the Python package for coverage:</p><pre>pip install pytest-cov</pre><p>You would get a coverage result and a coverage report like this:</p><pre>pytest --cov=mlops tests_folder</pre><h4>A simple test case</h4><p>Starting with a simple function that loads in a config yaml file and has type hinting:</p><pre>import yaml<br><br><br>def load_config(config_path: str) -&gt; dict:<br>    &quot;&quot;&quot;Function to load config file.&quot;&quot;&quot;<br>    cfg = []<br>    with open(config_path, &quot;r&quot;) as ymlfile:<br>        cfg = yaml.safe_load(ymlfile)<br>    return cfg</pre><p><strong>Codium AI comes up with the following test cases:</strong></p><ul><li>Check if the data is loaded properly</li><li>Check if the type of the loaded object is a dict</li><li>Check if it handles empty files</li><li>Check if it handles not existing files</li><li>Check if it handles invalid yaml files</li><li>Check if it handles invalid inputs</li></ul><p>This is quite interesting. Codium AI does not only give me tests I can immediately implement in my code base but also tells me what kind of tests I should implement.</p><p>With one click I also get an entire file for testing with import statements (based on the root folder).</p><pre># Generated by CodiumAI<br>from src.mlops.utils import load_config<br>import pytest<br><br>class TestLoadConfig:<br><br>    # Loads a valid YAML config file successfully<br>    def test_load_valid_yaml_config_file(self):<br>        # Arrange<br>        config_path = &quot;valid_config.yaml&quot;<br>    <br>        # Act<br>        result = load_config(config_path)<br>    <br>        # Assert<br>        assert isinstance(result, dict)<br><br>    # Returns a dictionary object containing the loaded config<br>    def test_return_loaded_config(self):<br>        # Arrange<br>        config_path = &quot;valid_config.yaml&quot;<br>    <br>        # Act<br>        result = load_config(config_path)<br>    <br>        # Assert<br>        assert isinstance(result, dict)<br><br>    # Handles empty config files gracefully<br>    def test_handle_empty_config_files(self):<br>        # Arrange<br>        config_path = &quot;empty_config.yaml&quot;<br>    <br>        # Act<br>        result = load_config(config_path)<br>    <br>        # Assert<br>        assert isinstance(result, dict)<br><br>    # Raises a FileNotFoundError if the config file path is invalid<br>    def test_raise_file_not_found_error(self):<br>        # Arrange<br>        config_path = &quot;invalid_path.yaml&quot;<br>    <br>        # Act and Assert<br>        with pytest.raises(FileNotFoundError):<br>            load_config(config_path)<br><br>    # Raises a yaml.parser.ParserError if the config file is not a valid YAML file<br>    def test_raise_parser_error(self):<br>        # Arrange<br>        config_path = &quot;invalid_yaml_config.yaml&quot;<br>    <br>        # Act and Assert<br>        with pytest.raises(yaml.parser.ParserError):<br>            load_config(config_path)<br><br>    # Raises a TypeError if the config file is not a string<br>    def test_raise_type_error(self):<br>        # Arrange<br>        config_path = 123<br>    <br>        # Act and Assert<br>        with pytest.raises(TypeError):<br>            load_config(config_path)</pre><p><strong>A few remarks:</strong></p><ul><li>the<strong> yaml </strong>import is missing here</li><li>the files defined in the test case do not exist yet</li><li>it uses the .yaml ending which is different from .yml what I use</li></ul><p>Great! So let’s take all of these, fix the missing import, create the missing invalid and empty files and see what happens when I run pytest.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x8icoeqk5pwEkQqGjuoBrQ.png" /></figure><p><strong>I am getting the following 3 errors:</strong></p><ul><li>test_handle_empty_config_files — assert False</li><li>test_raise_parser_error — Failed: DID NOT RAISE &lt;class ‘yaml.parser.ParserError’&gt;</li><li>test_raise_type_error — OSError: [WinError 6] The handle is invalid</li></ul><p>These errors tell me that I can still improve my code base so that it covers more cases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/290/1*KlK01EDYtK1bIBQ6C7p3MQ.png" /></figure><p>After some grinding I got all tests to pass and even found a bug in my code:</p><pre>def load_config(config_path: str) -&gt; dict:<br>    &quot;&quot;&quot;Function to load config file.&quot;&quot;&quot;<br>    cfg = {} # This is a dict now <br>    if type(config_path) is str:<br>        with open(config_path, &quot;r&quot;) as ymlfile:<br>            cfg = yaml.safe_load(ymlfile)<br>        if cfg is None: <br>            cfg = {}<br>        elif type(cfg) is not dict:<br>            raise yaml.parser.ParserError<br>    else: <br>        raise TypeError<br>    return cfg</pre><p>I don’t really like the code as it is right now so I asked <a href="https://www.phind.com/search?home=true"><strong>phind</strong></a> (another VSCode extension) to improve it:</p><pre>def load_config(config_path: str) -&gt; dict:<br>   &quot;&quot;&quot;Function to load config file.&quot;&quot;&quot;<br>   cfg = {}<br>   if isinstance(config_path, str):<br>       try:<br>           with open(config_path, &quot;r&quot;) as ymlfile:<br>               cfg = yaml.safe_load(ymlfile)<br>           if cfg is None: <br>              cfg = {}<br>           elif not isinstance(cfg, dict):<br>              raise yaml.parser.ParserError(&quot;Loaded configuration is not a dictionary.&quot;)<br>       except FileNotFoundError:<br>           raise FileNotFoundError(f&quot;File {config_path} not found.&quot;)<br>   else: <br>       raise TypeError(&quot;Input must be a string.&quot;)<br>   return cfg</pre><p>The tests are still passing.</p><h4>Coverage</h4><p>The final step is to look at the coverage report:</p><pre>mlops\utils.py         20      0   100%</pre><p>Looking good!</p><p>I did comment out all the other code in the file just for this example. The load_config function is fully covered by the tests.</p><h3>Summary</h3><p>We looked at the CodiumAI extension using VSCode and created unit tests for a Python function. We saw that CodiumAI not only provides unit tests but also looks at different test cases like invalid file formats or invalid input. After updating the function so that all tests would pass we asked phind to improve our code base. We checked if all tests are still running. Finally we looked at our code coverage.</p><blockquote>If you found this article useful, clap, comment and please follow me.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ba68c1234c97" width="1" height="1" alt=""><hr><p><a href="https://towardsdev.com/codium-ai-writing-python-tests-with-an-ai-assistant-part-1-ba68c1234c97">Codium AI — Writing Python tests with an AI assistant — Part 1</a> was originally published in <a href="https://towardsdev.com/">Towards Dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Re-creating an AWS inference architecture based on Sagemaker and Airflow in Azure]]></title>
            <link>https://medium.com/towards-data-engineering/re-creating-an-aws-inference-architecture-based-on-sagemaker-and-airflow-in-azure-977a65c76c21?source=rss-b354350522f5------2</link>
            <guid isPermaLink="false">https://medium.com/p/977a65c76c21</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[mlops]]></category>
            <dc:creator><![CDATA[DataFairy]]></dc:creator>
            <pubDate>Mon, 11 Dec 2023 10:18:05 GMT</pubDate>
            <atom:updated>2023-12-21T11:50:16.772Z</atom:updated>
            <content:encoded><![CDATA[<h4>How I copied an AWS MLOPS architecture and transformed it to work in Azure using Airflow and Azure ML.</h4><h3>Introduction</h3><p>About two months ago I got asked the question if it is possible to run an inference pipeline using Azure Managed Airflow and Azure Machine Learning Studio. I had never attempted that before and got curious about a possible solution. After a short google search (yes, I still do that ;) I came across a medium post where they build exactly what I was looking for but in AWS. I had a general answer showing that it is possible to build an inference pipeline using Apache Airflow and a ML tool. But can I do the same in Azure? Are the tools available and how well can they talk to each other? Let’s find out.</p><h3>The original setup</h3><p>In the original architecture they used various AWS tools:</p><ul><li>Storage bucket</li><li>AWS SageMaker</li><li>SQS Queue</li><li>AWS Eventbridge</li><li>ML Compute Instance</li><li>Airflow installed on an EC2 instance</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*rf7dDme8m8ZF27V5.jpeg" /><figcaption>Image reference below.</figcaption></figure><p>They used a Kaggle competition dataset with Tweets data and the HuggingFace BERT model. For my use case I will reuse an existing model I had already registered and the sample requests from a previous article for simplicity. The input data and model shouldn’t make much of a difference.</p><h3>Azure Setup</h3><p>For the setup in Azure I will be using the following tools:</p><ul><li>Managed Airflow/local Airflow instance</li><li>Blob Storage</li><li>Azure ML Studio</li><li>Compute Instance</li><li>Azure Service Bus</li></ul><h4>Architecture in Azure</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m8mWfEuPjMlRxaDRnQ0E1Q.png" /></figure><p>I am reusing a previous Airflow pipeline for inference where I create and clean up the compute instance in Azure ML.</p><p><a href="https://medium.com/towards-data-engineering/calling-an-azure-ml-endpoint-using-apache-airflow-and-the-azure-ml-provider-package-802e8c0d0623">Calling an Azure ML endpoint using Apache Airflow and the Azure ML provider package</a></p><p>I will be using <strong>Pydantic</strong> to load, clean and prepare the data. That’s just code I have ready to go. It can be done much simpler but I prefer to show you the solution I would implement myself.</p><p><a href="https://medium.com/towards-data-engineering/pydantic-airflow-data-validation-in-python-and-the-cloud-9fcdf4551a4d">Pydantic &amp; Airflow — Data Validation in Python and the cloud</a></p><h4>Prerequisites:</h4><ul><li>A previously registered model and an Azure ML resource.</li><li>New requests are made available to Airflow using a connection to blob storage from the admin panel.</li></ul><ul><li><a href="https://medium.com/@datafairy/deploying-a-machine-learning-model-in-azure-ml-80c5dec152a3">Deploying a machine learning model in Azure ML</a></li><li><a href="https://medium.com/@datafairy/airflow-and-azure-storage-accounts-4db1c475b85d">Airflow and Azure Storage Accounts</a></li></ul><h4>The process:</h4><ul><li>The data is loaded and cleaned in Airflow and then pushed to a specific location on blob storage.</li><li>While the requests are prepared the compute instance is created.</li><li>Next the inference job is triggered by Airflow to run on Azure ML.</li><li>The job takes the registered model and the new request data as inputs.</li><li>After every input dataset has been scored the job sends the result to Azure Service Bus.</li><li>Once the job has succeeded Airflow automatically moves on to the next step. No need to track the status of the job for now.</li><li>In the next step the messages are read from the queue.</li><li>At the same time all resources created by Airflow are removed again.</li></ul><h4>Remarks:</h4><p>In the above example I am using Azure Service Bus to track the job status and to get the prediction results. The Airflow task that is running the job is also tracking the job status. I don’t need the results of Service Bus for that. Nevertheless I thought it was an interesting challenge to use Azure Service Bus. I decided to send the predictions as messages to the queue and to retrieve them in the following step.</p><p><strong>Various ways to send messages to Azure Service Bus with Python:</strong></p><pre>#servicebus.py<br><br>from typing import List<br>import json<br>from azure.servicebus.aio import ServiceBusClient<br>from azure.servicebus import ServiceBusMessage<br>from azure.identity.aio import DefaultAzureCredential<br><br><br>async def send(sb_name: str, queue_name: str, type: str, credential: DefaultAzureCredential, messages: List[dict]):<br>    &quot;&quot;&quot;Send messages to the Service Bus queue.&quot;&quot;&quot;<br>    async with ServiceBusClient(<br>        fully_qualified_namespace=f&quot;sb://{sb_name}.servicebus.windows.net&quot;,<br>        credential=credential,<br>        logging_enable=True) as servicebus_client:<br>        sender = servicebus_client.get_queue_sender(queue_name=queue_name)<br>        async with sender:<br>            if type == &quot;single&quot;:<br>                await send_single_message(sender, messages[0])<br>            elif type == &quot;list&quot;:<br>                await send_a_list_of_messages(sender, messages)<br>            elif type == &quot;batch&quot;:<br>                await send_batch_message(sender, messages)<br>            else:<br>                raise ValueError(&quot;Invalid type&quot;)<br>        await credential.close()<br><br><br>async def send_single_message(sender, single_message: dict):<br>    &quot;&quot;&quot;Create a Service Bus message and send it to the queue&quot;&quot;&quot;<br>    message = ServiceBusMessage(json.dumps(single_message))<br>    await sender.send_messages(message)<br><br><br>async def send_a_list_of_messages(sender, message_list: List[dict]):<br>    &quot;&quot;&quot;Create a list of messages and send it to the queue&quot;&quot;&quot;<br>    messages = [ServiceBusMessage(json.dumps(message)) for message in message_list]<br>    await sender.send_messages(messages)<br><br><br>async def send_batch_message(sender, message_list: List[dict]):<br>    &quot;&quot;&quot;Create a batch of messages&quot;&quot;&quot;<br>    async with sender:<br>        batch_message = await sender.create_message_batch()<br>        for message in message_list:<br>            try:<br>                batch_message.add_message(ServiceBusMessage(json.dumps(message)))<br>            except ValueError:<br>                # ServiceBusMessageBatch object reaches max_size.<br>                # New ServiceBusMessageBatch object can be created here to send more data.<br>                break<br>        await sender.send_messages(batch_message)</pre><h4>Azure Service Bus Airflow Connection:</h4><p>The Airflow tasks for Azure Service Bus do not accept connection id’s for Service Bus. You still need to make a connection in the admin panel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/699/1*jwwIlovd__ehkY_IGCwhVg.png" /></figure><h4>The pipeline:</h4><pre>import os<br>import pendulum<br>from typing import List<br>import json<br>from inference.etl import load_data, prepare_requests, clean_request<br>from inference.request import Request<br><br>from airflow import DAG<br>from airflow.operators.empty import EmptyOperator<br>from azure.ai.ml import Input, command<br>from azure.ai.ml.entities import ComputeInstance<br>from azure.ai.ml.constants import AssetTypes, InputOutputModes<br>from airflow.models.baseoperator import chain<br>from airflow.decorators import task<br>from airflow.operators.python import PythonOperator<br>from airflow.providers.microsoft.azure.operators.asb import (<br>    AzureServiceBusReceiveMessageOperator,<br>)<br>from airflow_provider_azure_machinelearning.operators.machine_learning.job import (<br>    AzureMachineLearningCreateJobOperator,<br>)<br>from airflow_provider_azure_machinelearning.operators.machine_learning.compute import (<br>    AzureMachineLearningCreateComputeResourceOperator,<br>    AzureMachineLearningDeleteComputeResourceOperator,<br>)<br><br>with DAG(<br>    dag_id=&quot;AML_predict_job&quot;,<br>    schedule=None,<br>    start_date=pendulum.datetime(2023, 1, 1, tz=&quot;UTC&quot;),<br>    catchup=False,<br>    schedule_interval=&quot;0 17 * * *&quot;,<br>    tags=[&quot;AML predict&quot;],<br>     default_args={<br>      # connection id for the service bus is defined in the dag<br>        &quot;azure_service_bus_conn_id&quot;: &quot;azure-sb-conn&quot;,<br>    },<br>) as dag:<br>    # set the connection to the Azure ML workspace and provide the model information<br>    connection_id = &quot;azure-ml-ws-conn&quot;<br>    model_info = &quot;credit-default-model:1&quot;<br>    queue_name = &quot;test_queue&quot;<br>    conda_environment = &quot;environ:version&quot;<br><br>    def get_list_of_raw_data() -&gt; List[str]:<br>        &quot;&quot;&quot;<br>        #### Dummy list of raw data. <br>        &quot;&quot;&quot;<br>        return [<br>            &#39;{&quot;input_data&quot;: {&quot;columns&quot;: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22],&quot;index&quot;: [0, 1],&quot;data&quot;: [[20000,2,2,1,24,2,2,-1,-1,-2,-2,3913,3102,689,0,0,0,0,689,0,0,0,0],[10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 10, 9, 8]]}}&#39;,<br>            &#39;{&quot;input_data&quot;: {&quot;columns&quot;: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22],&quot;index&quot;: [0, 1],&quot;data&quot;: [[3913,3102,689,0,0,0,0,689,0,0,0,0,20000,2,2,1,24,2,2,-1,-1,-2,-2],[10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 10, 9, 8]]}}&#39;,<br>            &#39;{&quot;input_data&quot;: {&quot;columns&quot;: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22],&quot;index&quot;: [0, 1],&quot;data&quot;: [[3913,3102,-689,0,0,0,0,-689,0,0,0,0,20000,2,2,1,-24,2,-2,1,1,2,2],[10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 10, 9, 8]]}}&#39;,<br>        ]<br>    <br>    def load_data_task() -&gt; List[Request]:<br>        &quot;&quot;&quot;<br>        #### Extract task<br>        &quot;&quot;&quot;<br>        ordered_data = load_data(get_list_of_raw_data())<br><br>        requests: List[Request] = [Request(**item) for item in ordered_data]<br><br>        return requests<br><br>    def clean_data_task(task_instance, **kwargs) -&gt; List[str]:<br>        &quot;&quot;&quot;<br>        #### Clean data task<br>        &quot;&quot;&quot;<br>        cleaned_requests = []<br><br>        requests = task_instance.xcom_pull(task_ids=&quot;load_data_task&quot;)<br><br>        for request in requests:<br>            cleaned_requests.append(json.dumps(clean_request(request)))<br><br>        return cleaned_requests<br><br>    def prepare_requests_task(task_instance, **kwargs):<br>        &quot;&quot;&quot;<br>        #### Load task: load cleaned data into database<br>        &quot;&quot;&quot;<br>        cleaned_requests = task_instance.xcom_pull(task_ids=&quot;clean_data_task&quot;)<br><br>        request_locations = prepare_requests(cleaned_requests)<br><br>        return request_locations<br>    <br>    load_data_task = PythonOperator(task_id=&quot;load_data_task&quot;, python_callable=load_data_task)<br>    clean_data_task = PythonOperator(task_id=&quot;clean_data_task&quot;, python_callable=clean_data_task)<br>    prepare_requests_task = PythonOperator(task_id=&quot;prepare_requests_task&quot;, python_callable=prepare_requests_task)<br><br>    compute1 = ComputeInstance(<br>        name=&quot;compute-321-inference&quot;,<br>        size=&quot;Standard_D2s_v3&quot;,  # spec for a compute instance<br>    )<br><br>    amlc_create_1 = AzureMachineLearningCreateComputeResourceOperator(<br>        task_id=&quot;create_compute_instance&quot;,<br>        conn_id=connection_id,<br>        compute=compute1,<br>        waiting=True,<br>    )<br><br>    amlc_delete_1 = AzureMachineLearningDeleteComputeResourceOperator(<br>        task_id=&quot;delete_compute_instance&quot;,<br>        conn_id=connection_id,<br>        compute_name=compute1.name,<br>    )<br>    # Service principal will need Azure Service Bus Data Owner role<br>    # Fill in the information about the service principal below<br>    env_servicebus = {<br>        &quot;AZURE_CLIENT_SECRET&quot;: &quot;&quot;,<br>        &quot;AZURE_TENANT_ID&quot;: &quot;&quot;,<br>        &quot;AZURE_CLIENT_ID&quot;: &quot;&quot;,<br>    }<br>    # define the path to the /src folder and the main.py file<br>    curr_dir = os.path.dirname(os.path.abspath(__file__))<br>    code_file_path = os.path.join(curr_dir, &quot;/opt/airflow/jobs/predict/src&quot;)<br><br>    predict_command_job = command(<br>        code=code_file_path,<br>        command=&quot;python main.py --input_data_folder ${{inputs.input_data_folder}} --input_config_yaml ${{inputs.input_config_yaml}}&quot;,<br>        environment=conda_environment,<br>        inputs={<br>            &quot;input_config_yaml&quot;: Input(<br>                mode=InputOutputModes.RO_MOUNT,<br>                type=AssetTypes.URI_FILE,<br>                path=&quot;azureml://datastores/xxx/paths/credit_defaults_model/config/config.yml&quot;,<br>            ),<br>            &quot;input_data_folder&quot;: Input(<br>                mode=InputOutputModes.RO_MOUNT,<br>                type=AssetTypes.URI_FOLDER,<br>                path=&quot;azureml://datastores/xxx/paths/inference_input&quot;,<br>            ),<br>        },<br>        environment_variables=env_servicebus,<br>        compute=compute1.name,<br>        display_name=&quot;predict_from_airflow&quot;,<br>        experiment_name=&quot;testing-airflow&quot;,<br>        description=&quot;predict command job&quot;,<br>    )<br>    predict_task = AzureMachineLearningCreateJobOperator(<br>        task_id=&quot;predict&quot;,<br>        job=predict_command_job,<br>        waiting=True,<br>        conn_id=connection_id,<br>    )<br><br>    check_job_status =  AzureServiceBusReceiveMessageOperator(<br>    task_id=&quot;receive_message_service_bus_queue&quot;,<br>    queue_name=queue_name,<br>    max_message_count=20,<br>    max_wait_time=5,<br>    )<br><br>    start_task = EmptyOperator(task_id=&quot;start&quot;)<br>    success_task = EmptyOperator(task_id=&quot;success&quot;)<br><br>    start_task &gt;&gt;  amlc_create_1 &gt;&gt; predict_task &gt;&gt; [amlc_delete_1, check_job_status] &gt;&gt; success_task<br>    start_task &gt;&gt; load_data_task &gt;&gt; clean_data_task &gt;&gt; prepare_requests_task &gt;&gt; predict_task</pre><h3>Results:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yU_QcFLUfYgDcBJ3Rvz5rQ.png" /></figure><p><strong>Service Bus Task output:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/328/1*ZMMafUJKVha-xHdQMWSHBQ.png" /><figcaption>Airflow logs with service bus messages.</figcaption></figure><h3>Summary:</h3><p>In this article we looked at how to create an inference pipeline using MLOPS in Azure. For this purpose we used a setup in AWS as inspiration. We then defined the components used in AWS and recreated a similar infrastructure in Azure using Airflow, Azure ML and Azure Service Bus. We used Pydantic for data validation and checked our predictions by reading the message from the service bus queue.</p><h4>Github:</h4><p><a href="https://github.com/datafairy-azure/inference">datafairy-azure/inference: Code for inference pipelines un airflow (github.com)</a></p><p><strong>AWS Setup:</strong></p><p><a href="https://medium.com/mlearning-ai/accelerate-ml-inference-data-pipeline-using-sagemaker-and-apache-airflow-f19207e896ca">Automate and streamline our ML inference pipeline with SageMaker and Airflow | MLearning.ai (medium.com)</a></p><h3>Note</h3><p><em>I am still experimenting a lot with Apache Airflow and Azure ML and I would really appreciate your feedback. I am also very much interested in other approaches, questions or ideas on how to improve the workflows I show here. Let me know what you think!</em></p><blockquote><em>If you found this article useful, please follow me.</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=977a65c76c21" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-data-engineering/re-creating-an-aws-inference-architecture-based-on-sagemaker-and-airflow-in-azure-977a65c76c21">Re-creating an AWS inference architecture based on Sagemaker and Airflow in Azure</a> was originally published in <a href="https://medium.com/towards-data-engineering">Towards Data Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Calling an Azure ML endpoint using Apache Airflow and the Azure ML provider package]]></title>
            <link>https://medium.com/towards-data-engineering/calling-an-azure-ml-endpoint-using-apache-airflow-and-the-azure-ml-provider-package-802e8c0d0623?source=rss-b354350522f5------2</link>
            <guid isPermaLink="false">https://medium.com/p/802e8c0d0623</guid>
            <category><![CDATA[azure-machine-learning]]></category>
            <category><![CDATA[apache-airflow]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[httpx]]></category>
            <dc:creator><![CDATA[DataFairy]]></dc:creator>
            <pubDate>Thu, 30 Nov 2023 12:34:46 GMT</pubDate>
            <atom:updated>2023-12-03T21:43:02.531Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to score your Machine Learning model that is deployed to an Azure ML endpoint using Apache Airflow and an Azure ML job.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6rDatHYGkhYevPE4UCCZ2A.png" /></figure><h3>What this article is about</h3><p>In this article we will create another pipeline to call an Azure ML endpoint using httpx and Apache Airflow. This time we will call the endpoint from Azure ML itself. We will look at the advantages of this method and how to get an Azure ML job started from Apache Airflow.</p><p><strong>Checkout my previous take on this topic:</strong></p><p><a href="https://medium.com/@datafairy/how-to-call-an-azure-ml-endpoint-using-apache-airflow-and-httpx-cda8d865ed9c">How to call an Azure ML endpoint using Apache Airflow and httpx</a></p><h3>Running predictions on AzureML</h3><p>Using Azure ML has a few advantages:</p><ul><li>Airflow will be used as intended and only orchestrates the process</li><li>Setting up a custom Python environment is straightforward</li><li>Data is mounted in Azure ML (read-write goes much faster)</li><li>Authentication is much simpler</li></ul><h4>Prerequisites:</h4><ul><li>Datastore is mounted with sufficient access rights (SAS token, Access key). use the key if you get a stream authentication error</li><li>Service principal to create Azure ML connection in Airflow</li><li>Azure ML endpoint with registered model deployed and accepting traffic</li><li>Test data</li></ul><ul><li><a href="https://medium.com/@datafairy/creating-a-mlops-pipeline-with-azure-machine-learning-service-b960fb80a7d1">Creating a MLOPS pipeline with Azure Machine Learning Service</a></li><li><a href="https://medium.com/@marthalasia/deploying-a-machine-learning-model-in-azure-ml-80c5dec152a3">Deploying a machine learning model in Azure ML</a></li></ul><h3>The Airflow-Azure ML pipeline</h3><p>The following pipeline consists of these steps:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1ARRHeQPZX3MrHWMJX1CGw.png" /></figure><ul><li><strong>Start</strong></li><li><strong>Create compute instance:</strong> creates a new compute instance using the Airflow Azure ML connection</li><li><strong>predict: </strong>Run the predict job on Azure ML</li><li><strong>Delete compute instance:</strong> deletes the compute instance using the Airflow Azure ML connection</li><li><strong>Success</strong></li></ul><pre>import os<br>import pendulum<br>from airflow import DAG<br>from airflow.operators.empty import EmptyOperator<br>from azure.ai.ml import Input, command<br>from azure.ai.ml.entities import AmlCompute, ComputeInstance<br>from azure.ai.ml.constants import AssetTypes, InputOutputModes<br><br>from airflow_provider_azure_machinelearning.operators.machine_learning.job import (<br>    AzureMachineLearningCreateJobOperator,<br>)<br>from airflow_provider_azure_machinelearning.operators.machine_learning.compute import (<br>    AzureMachineLearningCreateComputeResourceOperator,<br>    AzureMachineLearningDeleteComputeResourceOperator,<br>)<br><br># Pipeline definition<br>with DAG(<br>    dag_id=&quot;AML_predict_job&quot;,<br>    schedule=None,<br>    start_date=pendulum.datetime(2023, 1, 1, tz=&quot;UTC&quot;),<br>    catchup=False,<br>    schedule_interval=&quot;0 17 * * *&quot;,<br>    tags=[&quot;AML predict&quot;],<br>) as dag:<br># Create an Azure ML Airflow connection named azure-ml-ws-conn<br>    connection_id = &quot;azure-ml-ws-conn&quot;<br> <br># Define the compute instance<br>    compute1 = ComputeInstance(<br>        name=&quot;af-test-instance&quot;,<br>        size=&quot;Standard_D2s_v3&quot;,  # spec for a compute instance<br>    )<br><br># Task to create the compute instance<br>    amlc_create_1 = AzureMachineLearningCreateComputeResourceOperator(<br>        task_id=&quot;create_compute_instance&quot;,<br>        conn_id=connection_id,<br>        compute=compute1,<br>        waiting=True,<br>    )<br># Task to delete the compute instance<br>    amlc_delete_1 = AzureMachineLearningDeleteComputeResourceOperator(<br>        task_id=&quot;delete_compute_instance&quot;,<br>        conn_id=connection_id,<br>        compute_name=compute1.name,<br>    )<br>    <br># Defining the location of the source code<br>    curr_dir = os.path.dirname(os.path.abspath(__file__))<br>    code_file_path = os.path.join(curr_dir, &quot;/opt/airflow/jobs/predict/src&quot;)<br><br># Defintion of the predict job with input parameters<br>    predict_command_job = command(<br>        code=code_file_path,<br>        command=&quot;python main.py --input_data_folder ${{inputs.input_data_folder}} --input_config_yaml ${{inputs.input_config_yaml}}&quot;,<br>        environment=&quot;credit-default-env:version_number&quot;,<br>        inputs={<br>            &quot;input_config_yaml&quot;: Input(<br>                mode=InputOutputModes.RO_MOUNT,<br>                type=AssetTypes.URI_FILE,<br>                # path to the config file on the mounted blob storage<br>                path=&quot;azureml://datastores/xxx/paths/credit_defaults_model/config/config.yml&quot;,<br>            ),<br>            &quot;input_data_folder&quot;: Input(<br>                mode=InputOutputModes.RO_MOUNT,<br>                type=AssetTypes.URI_Folder,<br>                # path to the sample request folder on the mounted blob storage<br>                path=&quot;azureml://datastores/xxx/paths/credit_defaults_model/data&quot;,<br>            ),<br>        },<br>        compute=compute1.name,<br>        display_name=&quot;predict_from_airflow&quot;,<br>        experiment_name=&quot;testing-airflow&quot;,<br>        description=&quot;predict command job&quot;,<br>    )<br><br># Job creation task <br>    predict_task = AzureMachineLearningCreateJobOperator(<br>        task_id=&quot;predict&quot;,<br>        job=predict_command_job,<br>        waiting=True,<br>        conn_id=connection_id,<br>    )<br><br>    start_task = EmptyOperator(task_id=&quot;start&quot;)<br>    success_task = EmptyOperator(task_id=&quot;success&quot;)<br><br>    start_task &gt;&gt; amlc_create_1 &gt;&gt; predict_task &gt;&gt; amlc_delete_1 &gt;&gt; success_task</pre><p>This example depends on the <a href="https://pypi.org/project/airflow-provider-azure-machinelearning/">airflow-provider-azure-machinelearning · PyPI</a> package. This is still an early version and bugs can be expected.</p><h4>The job definition:</h4><p>Besides the job definition the other steps in the pipeline are pretty straightforward.</p><pre># Defining the location of the source code<br>    curr_dir = os.path.dirname(os.path.abspath(__file__))<br>    code_file_path = os.path.join(curr_dir, &quot;/opt/airflow/jobs/predict/src&quot;)<br><br># Defintion of the predict job with input parameters<br>    predict_command_job = command(<br>        code=code_file_path,<br>        command=&quot;python main.py --endpoint ${{inputs.endpoint}} --input_config_yaml ${{inputs.input_config_yaml}}&quot;,<br>        environment=&quot;credit-default-env:version_number&quot;,<br>        inputs={<br>            &quot;input_config_yaml&quot;: Input(<br>                mode=InputOutputModes.RO_MOUNT,<br>                type=AssetTypes.URI_FILE,<br>                path=&quot;azureml://datastores/xxx/paths/credit_defaults_model/config/config.yml&quot;,<br>            ),<br>            &quot;input_data_folder&quot;: Input(<br>                mode=InputOutputModes.RO_MOUNT,<br>                type=AssetTypes.URI_Folder,<br>                path=&quot;azureml://datastores/xxx/paths/credit_defaults_model/data&quot;,<br>            )<br>        },<br>        compute=compute1.name,<br>        display_name=&quot;predict_from_airflow&quot;,<br>        experiment_name=&quot;testing-airflow&quot;,<br>        description=&quot;predict command job&quot;,<br>    )</pre><p>The Airflow job definition is very much similar to the job definition in Azure ML. One difference is that we have to make sure Airflow can find the source files we want to provide to run the job (jobs/predict/src).</p><pre>curr_dir = os.path.dirname(os.path.abspath(__file__))<br>code_file_path = os.path.join(curr_dir, &quot;/opt/airflow/jobs/predict/src&quot;)</pre><p>They should have been made available to Airflow in the Dockerfile:</p><pre>FROM quay.io/astronomer/astro-runtime:9.1.0<br><br>COPY requirements.txt /opt/airflow/requirements.txt<br><br>COPY dags/jobs /opt/airflow/jobs</pre><h4><strong>Authorization:</strong></h4><p>According to the <a href="https://github.com/Azure/airflow-provider-azure-machinelearning">Azure github documentation</a> you need a service principal with contributor rights on the resource group level and, digging deeper, also Storage Blob Data Reader on the storage account. I found that not sufficient. I got a <strong>StreamAccessAuthentication</strong> error with this setup. The solution for me was to mount the blob storage on Azure ML using an access key and not just a SAS token.</p><h4>Main.py</h4><pre>import argparse<br>import sys, os<br><br>sys.path.append(os.path.dirname(os.path.realpath(__file__)))<br>from inference.utils import (<br>    define_headers,<br>    load_config,<br>    call_endpoint_with_requests,<br>    parse_requests,<br>)<br><br><br>def parse_args():<br>    &quot;&quot;&quot;Parse input arguments&quot;&quot;&quot;<br>    parser = argparse.ArgumentParser()<br>    parser.add_argument(<br>        &quot;--input_config_yaml&quot;, type=str, help=&quot;config location on azure&quot;<br>    )<br>    parser.add_argument(<br>        &quot;--input_data_folder&quot;, type=str, help=&quot;data location on azure&quot;<br>    )<br>    return parser.parse_args()<br><br><br>def main(args):<br>    cfg = load_config(args.input_config_yaml)<br>    headers = define_headers(cfg)<br><br>    # Define the location of the sample data<br>    paths = [<br>        args.input_data_folder + &quot;sample_request_1.json&quot;,<br>        args.input_data_folder + &quot;sample_request_2.json&quot;,<br>    ]<br><br>    # Loading in the requests from json files <br>    request_items = parse_requests(paths)<br><br>    return call_endpoint_with_requests(<br>        request_items, headers, cfg, &quot;httpx&quot;<br>    )<br><br>if __name__ == &quot;__main__&quot;:<br>    args = parse_args()<br><br>    response = main(args)<br>    print(response)</pre><h4>Github Link:</h4><p><a href="https://github.com/datafairy-azure/inference">datafairy-azure/inference: Code for inference pipelines un airflow (github.com)</a></p><h3>Summary</h3><p>In this article we looked at how to run Azure ML jobs using Apache Airflow. We created an Airflow pipeline that creates and deletes compute instances in Azure ML and runs jobs. We accessed data from blob storage mounted on Azure ML. Finally we ran a predict job by calling an Azure ML endpoint with sample requests.</p><h3>Note</h3><p><em>I am still experimenting a lot with Apache Airflow and Azure ML and I would really appreciate your feedback. I am also very much interested in other approaches, questions or ideas on how to improve the workflows I show here. Let me know what you think!</em></p><blockquote><em>If you found this article useful, please follow me.</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=802e8c0d0623" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-data-engineering/calling-an-azure-ml-endpoint-using-apache-airflow-and-the-azure-ml-provider-package-802e8c0d0623">Calling an Azure ML endpoint using Apache Airflow and the Azure ML provider package</a> was originally published in <a href="https://medium.com/towards-data-engineering">Towards Data Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to call an Azure ML endpoint using Apache Airflow and httpx]]></title>
            <link>https://towardsdev.com/how-to-call-an-azure-ml-endpoint-using-apache-airflow-and-httpx-cda8d865ed9c?source=rss-b354350522f5------2</link>
            <guid isPermaLink="false">https://medium.com/p/cda8d865ed9c</guid>
            <category><![CDATA[apache-airflow]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[azure-machine-learning]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[inference]]></category>
            <dc:creator><![CDATA[DataFairy]]></dc:creator>
            <pubDate>Sun, 26 Nov 2023 15:55:11 GMT</pubDate>
            <atom:updated>2023-11-27T19:28:33.874Z</atom:updated>
            <content:encoded><![CDATA[<h4>A detailed guide on how to run predictions on a registered model in Azure ML using Airflow and httpx.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m1JbtiE1Ad_teo0U6cpbrQ.png" /></figure><h3>What this article is about</h3><p>In this article we will call an existing Azure ML endpoint and generate predictions for data stored locally. A previously registered model is already deployed to the endpoint. At the end of this article we will have a code base to run predictions from Airflow using an Airflow instance.</p><p><a href="https://medium.com/@datafairy/creating-a-mlops-pipeline-with-azure-machine-learning-service-b960fb80a7d1">Creating a MLOPS pipeline with Azure Machine Learning Service</a></p><h4>Prerequisites:</h4><ul><li>Azure ML instance</li><li>service principal with contributor rights on the resource group</li><li>Airflow running locally</li><li>Endpoint with deployed registered model in Azure ML</li><li>sample data to send as requests to test your model</li><li>Azure ML environment</li></ul><p><a href="https://medium.com/@marthalasia/deploying-a-machine-learning-model-in-azure-ml-80c5dec152a3">Deploying a machine learning model in Azure ML</a></p><h4>Remark on endpoints:</h4><p>Make sure to update your authentication type to AML token. Default is set to key:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/591/1*pjKzsHROQW9CC91HFwK9jg.png" /></figure><h3>httpx</h3><blockquote>HTTPX is a fully featured HTTP client for Python 3, which provides sync and async APIs, and support for both HTTP/1.1 and HTTP/2.</blockquote><p>I came across httpx because it was featured as one of the “Python packages you need to know” list. It is also said to fix some of the issues that arise with the requests library (async requests). <a href="https://www.youtube.com/watch?v=o06MyVhYte4">15 Python Libraries You Should Know About in 2023 — YouTube</a></p><p>For this example I decided to give it a go. The code will work with the requests package as well though:</p><pre>res = requests.post(<br>      url=&quot;https://&quot; + endpoint + &quot;.&quot; + location + &quot;.inference.ml.azure.com/score&quot;,<br>      data=json.dumps(data),<br>      headers=headers,<br>      )<br>response.append({&quot;status&quot;: str(res.status_code), &quot;prediction&quot;: res.text})</pre><h4>Sample request:</h4><p>To run the code below you will need some sample data in the following format:</p><pre>{<br> &quot;input_data&quot;: {<br>  &quot;columns&quot;: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22],<br>  &quot;index&quot;: [0, 1],<br>  &quot;data&quot;: [[20000, 2, 2, 1, 24, 2, 2, -1, -1, -2, -2, 3913, 3102, 689, 0, 0, 0, 0, 689, 0, 0, 0, 0], [10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 10, 9, 8]]<br> }<br>}</pre><p>Changing the items after data slightly is sufficient for this example.</p><h4>Setting up Airflow:</h4><p>I am using the astronomer cli to run Airflow. For this example I installed a custom package called inference (dags/jobs/predict/src/inference) and mounted some data to Airflow (dags/data). This is how I updated the Dockerfile:</p><pre>FROM quay.io/astronomer/astro-runtime:9.1.0<br><br>COPY requirements.txt /opt/airflow/requirements.txt<br><br>COPY dags/data /opt/airflow/data<br><br>COPY dags/jobs /opt/airflow/jobs<br><br>USER root<br><br>RUN pip install -r /opt/airflow/requirements.txt<br><br>RUN pip install /opt/airflow/jobs/predict/src</pre><p>For this example you don’t need to create a custom inference package and can just use the utils.py functions directly in your pipeline.</p><h3>The pipeline: Airflow</h3><pre>from airflow.decorators import dag, task<br>import pendulum as pm<br><br>from inference.utils import (<br>     define_headers_with_sp, <br>     call_endpoint_with_requests<br>     parse_requests<br>)<br><br><br>@dag(<br>    schedule=None,<br>    start_date=pm.datetime(2023, 1, 1),<br>    catchup=False,<br>    default_args={<br>        &quot;retries&quot;: 2,<br>    },<br>    tags=[&quot;predict&quot;],<br>)<br>def predict_with_airflow():<br>    &quot;&quot;&quot;<br>    ### Predict pipeline<br>    &quot;&quot;&quot;<br><br>    @task()<br>    def trigger_inference_job():<br>        &quot;&quot;&quot;<br>        #### Calls an endpoint with the request data given<br>        &quot;&quot;&quot;<br>        <br>        # Using local request data mounted to opt/airflow/data<br>        paths = [<br>            &quot;/opt/airflow/data/sample_request_1.json&quot;,<br>            &quot;/opt/airflow/data/sample_request_2.json&quot;,<br>        ]<br>        <br>        # Defining the endpoint which I normally store in a config.yml file<br>        cfg = {<br>            &quot;deployments&quot;: {<br>                &quot;endpoint_name&quot;: &quot;endpoint-xxx&quot;<br>                },<br>            &quot;connections&quot;: {<br>                &quot;location&quot;: &quot;azure-location&quot;<br>                }<br>            }<br>        <br>        # Authentication headers <br>        headers = define_headers_with_sp(<br>            config_path=&quot;/opt/airflow/data/config.json&quot;, endpoint=cfg[&quot;deployments&quot;][&quot;endpoint_name&quot;]<br>        )<br>        <br>        # Loading in the requests from json files <br>        request_items = parse_requests(paths)<br><br>        print(&quot;Running inference job...using &quot; + &quot;httpx&quot;)<br>        <br>        # Calling the endpoint<br>        response = call_endpoint_with_requests(request_items, headers, cfg, &quot;httpx&quot;)<br><br>        return response<br><br>    print(trigger_inference_job())<br><br><br>predict_with_airflow()</pre><h4>Authentication</h4><p>The authentication to Azure ML is done using a service principal’s client id and secret. The connection is done using the MLClient library and the credentials are stored in the headers.</p><pre>headers = define_headers_with_sp(<br>            config_path=&quot;/opt/airflow/data/config.json&quot;, endpoint=endpoint<br>        )</pre><p>The custom function <strong>define_headers_with_sp</strong> is defined as follows (dags/jobs/predict/src/inference/utils.py):</p><pre># utils.py<br>from azure.ai.ml import MLClient<br>from azure.identity import DefaultAzureCredential<br>from typing import List<br>import requests<br>import logging<br>import json<br>from http import client as http_client<br><br><br>def call_endpoint_with_requests(<br>    request_items: List[str], headers: dict, cfg: dict, request_type: str = &quot;requests&quot;<br>) -&gt; List[str]:<br>    &quot;&quot;&quot;Calls the endpoint with the given request data.&quot;&quot;&quot;<br>    response = []<br><br>    define_logging_for_request_type(request_type)<br><br>    for item in request_items:<br>        res = requests.post(<br>            url=&quot;https://&quot;<br>            + cfg[&quot;deployments&quot;][&quot;endpoint_name&quot;]<br>            + cfg[&quot;connections&quot;][&quot;location&quot;]<br>            + &quot;.inference.ml.azure.com/score&quot;,<br>            data=item,<br>            headers=headers,<br>        )<br>        response.append({&quot;status&quot;: str(res.status_code), &quot;prediction&quot;: res.text})<br><br>    return response<br><br><br>def define_logging_for_request_type(request_type: str):<br>    http_client.HTTPConnection.debuglevel = 1<br>    if request_type == &quot;httpx&quot;:<br>        logging.basicConfig(<br>            format=&quot;%(levelname)s [%(asctime)s] %(name)s - %(message)s&quot;,<br>            datefmt=&quot;%Y-%m-%d %H:%M:%S&quot;,<br>            level=logging.DEBUG,<br>        )<br>    elif request_type == &quot;requests&quot;:<br>        logging.basicConfig()<br>        logging.getLogger().setLevel(logging.DEBUG)<br>        requests_log = logging.getLogger(&quot;requests.packages.urllib3&quot;)<br>        requests_log.setLevel(logging.DEBUG)<br>        requests_log.propagate = True<br><br><br>def parse_requests(paths: List[str]) -&gt; List[str]:<br>    &quot;&quot;&quot;<br>    Returns a list of request data items from a list for files paths.<br>    &quot;&quot;&quot;<br>    request_items = []<br>        for path in paths:<br>            with open(path, &quot;r&quot;) as file:<br>                data = json.load(file)<br>                request_items.append(json.dumps(data))<br>    return request_items<br><br><br>def define_headers_with_sp(endpoint: str, config_path=&quot;&quot;) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Creates headers for the http call to Azure ML using a configuration file.<br>    &quot;&quot;&quot;<br>    ml_client = ml_connect_with_sp(config_path)<br><br>    endpoint_cred = ml_client.online_endpoints.get_keys(<br>        name=endpoint<br>    ).access_token<br><br>    headers = {<br>        &quot;Authorization&quot;: f&quot;Bearer {endpoint_cred}&quot;,<br>        &quot;Content-type&quot;: &quot;application/json&quot;,<br>    }<br><br>    return headers<br><br><br>def ml_connect_with_sp(config_path: str) -&gt; MLClient:<br>    &quot;&quot;&quot;<br>    Returns the ML client either for a default credential or a configuration file.<br>    &quot;&quot;&quot;<br>    credential=DefaultAzureCredential()<br>    credential.get_token(&quot;https://management.azure.com/.default&quot;)<br><br>    try:<br>        ml_client = MLClient.from_config(credential=credential)<br>    except Exception:<br>        ml_client = MLClient.from_config(credential=credential, path=config_path)<br><br>    return ml_client</pre><p><strong>The configuration is as follows (dags/data/config.json):</strong></p><pre># config.json (does not work with yaml)<br>{<br>    &quot;subscription_id&quot;: &quot;xxx&quot;,<br>    &quot;resource_group&quot;: &quot;xxx&quot;,<br>    &quot;workspace_name&quot;: &quot;xxx&quot;<br>}</pre><h4>Logging:</h4><p>I included a custom function to display logging of the requests for both httpx and requests packages. This was very useful to get the pipeline running. The setup used the logging package and is quiet similar but slightly different for httpx and requests.</p><pre>def define_logging_for_request_type(request_type: str):<br>    http_client.HTTPConnection.debuglevel = 1<br>    if request_type == &quot;httpx&quot;:<br>        logging.basicConfig(<br>            format=&quot;%(levelname)s [%(asctime)s] %(name)s - %(message)s&quot;,<br>            datefmt=&quot;%Y-%m-%d %H:%M:%S&quot;,<br>            level=logging.DEBUG,<br>        )<br>    elif request_type == &quot;requests&quot;:<br>        logging.basicConfig()<br>        logging.getLogger().setLevel(logging.DEBUG)<br>        requests_log = logging.getLogger(&quot;requests.packages.urllib3&quot;)<br>        requests_log.setLevel(logging.DEBUG)<br>        requests_log.propagate = True</pre><p><strong>The logging output for httpx:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/1*MS3MqYRpvov7lSgcxHNG0A.png" /></figure><h4>Result:</h4><pre> Returned value was: [{&#39;status&#39;: &#39;200&#39;, &#39;prediction&#39;: &#39;[1, 0]&#39;}, {&#39;status&#39;: &#39;200&#39;, &#39;prediction&#39;: &#39;[1, 0]&#39;}]</pre><h3>Summary</h3><p>In this article we looked at how to run predictions in Azure ML using an existing endpoint and a registered model. We looked at the advantages of using httpx over the request package. We created a pipeline in Airflow and looked at the authentication requirements. Finally we looked at logging for both the requests and httpx package.</p><p><strong>Github link:</strong></p><p><a href="https://github.com/datafairy-azure/inference">datafairy-azure/inference: Code for inference pipelines un airflow (github.com)</a></p><h3>Note</h3><p><em>I am still experimenting a lot with Apache Airflow and Azure ML and I would really appreciate your feedback. I am also very much interested in other approaches, questions or ideas on how to improve the workflows I show here. Let me know what you think!</em></p><blockquote>If you found this article useful, please follow me.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cda8d865ed9c" width="1" height="1" alt=""><hr><p><a href="https://towardsdev.com/how-to-call-an-azure-ml-endpoint-using-apache-airflow-and-httpx-cda8d865ed9c">How to call an Azure ML endpoint using Apache Airflow and httpx</a> was originally published in <a href="https://towardsdev.com/">Towards Dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>