<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Matt Collins on Medium]]></title>
        <description><![CDATA[Stories by Matt Collins on Medium]]></description>
        <link>https://medium.com/@mc12338?source=rss-d1970f1605f1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*1kVDqrw7GBUhBC852gYBVQ.jpeg</url>
            <title>Stories by Matt Collins on Medium</title>
            <link>https://medium.com/@mc12338?source=rss-d1970f1605f1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 08:58:39 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mc12338/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Methods for generating synthetic descriptive data]]></title>
            <link>https://medium.com/data-science/methods-for-generating-synthetic-descriptive-data-c6678cc10aff?source=rss-d1970f1605f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/c6678cc10aff</guid>
            <category><![CDATA[synthetic-data]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-modelling]]></category>
            <category><![CDATA[pyspark]]></category>
            <dc:creator><![CDATA[Matt Collins]]></dc:creator>
            <pubDate>Thu, 04 Jan 2024 19:10:16 GMT</pubDate>
            <atom:updated>2024-01-04T19:10:16.159Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Generating Synthetic Descriptive Data in PySpark</strong></h3><h4>Use various data source types to quickly generate text data for artificial datasets.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UrKmd4o1A0aMje6JfR5o4w.jpeg" /><figcaption>Image generated with DALL-E 3</figcaption></figure><p>In a <a href="https://medium.com/towards-data-science/create-many-to-one-relationships-between-columns-in-a-synthetic-table-with-pyspark-udfs-41e39d97c936">previous article</a>, we explored creating many-to-one relationships between columns in a synthetic PySpark DataFrame. This DataFrame only consisted of Foreign Key information and we didn’t produce any textual information that might be useful in a demo DataSet.</p><p>For anyone looking to populate an artificial dataset, it is likely you will want to produce descriptive data such as product information, location details, customer demographics, etc.</p><p>In this post, we’ll dig into a few sources that can be used to create synthetic text data at little effort and cost, and use the techniques to pull together a DataFrame containing customer details.</p><h3>Why create an synthetic dataset?</h3><p>Synthetic datasets are a great way to anonymously demonstrate your data product, such as a website or analytics platform. Allowing users and stakeholders to interact with example data, exposing meaningful analysis without breaching any privacy concerns with sensitive data.</p><p>It can also be great for exploring Machine Learning algorithms, allowing Data Scientists to train models in the case of limited real data.</p><p>Performance testing Data Engineering pipeline activities is another great use case for synthetic data, giving teams the ability to ramp up the scale of data pushed through an infrastructure and identify weaknesses in the design, as well as benchmarking runtimes.</p><p>In my case, I’m currently creating an example dataset to performance-test some Power BI capabilities at high volumes, which I’ll be writing about in due course.</p><p>The dataset will contain sales data, including transaction amounts and other descriptive features such as store location, employee name and customer email address.</p><h3>Random characters</h3><p>Starting off simple, we can use some built-in functionality to generate random text data. Importing the <em>random</em> and <em>string</em> Python modules, we can use the following simple function to create a random string of the desired length.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0a1fa401fa99f960c75550f720fc30a6/href">https://medium.com/media/0a1fa401fa99f960c75550f720fc30a6/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v2lo5IBU-MTsJ0hUbAxSsg.png" /><figcaption>Image by Author: Screenshot of several examples of results</figcaption></figure><p>We can run this multiple times to generate enough data for our column, appending the information to a list.</p><p>We will review how to add this to our DataFrame further down this post.</p><h4>Benefits and Limitations</h4><p>This kind of data generation is very generic, with limited applications in demo datasets. That being said, it can be combined with other string generation techniques (such as concatenation) to give a bit more value at very little effort.</p><p>This can be seen below for random usernames, with a base of a first name appended by an underscore and random characters:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9a477ad86d8c85b806b3a37217af5df6/href">https://medium.com/media/9a477ad86d8c85b806b3a37217af5df6/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6GFODQlt-OUoE6OM8BwJog.png" /><figcaption>Image by Author: Screenshot of several usernames created with random strings appended.</figcaption></figure><p>Example data points where this could be useful may include:</p><ul><li>Email addresses</li><li>Passwords</li><li>Product codes</li><li>Usernames</li></ul><h3>APIs</h3><p>APIs are a great source of information, and will likely be data sources when building your real Analytics platforms!</p><p>However, there are also many API endpoints worth querying for generic placeholder data that can give you more meaning for demo dashboards through representative data for various topics, such as currency rates.</p><p>If we want to retrieve geographical data for a field for a user’s country, we could generate this from the <a href="https://restcountries.com/">rest countries</a> API endpoint. This API is free to access and requires no sign-up to get started. Using the <em>requests</em> python module we’re up and running with a list of countries very quickly.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/64b36829129e6237796921eccb364af5/href">https://medium.com/media/64b36829129e6237796921eccb364af5/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*93srjCGIGHKzFKDBbeZUSQ.png" /><figcaption>Image by Author: print(sample_countries) statement showing a sample of the shuffled results.</figcaption></figure><p><strong><em>Note: You should always sense-check your outputs — this API request returns 250 results, which exceeds the real-life number of countries!</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iITJThV7JhyjCNW3b18Fcw.png" /><figcaption>Image by Author: Snippet of the pprint(sorted(countries_list)) statement, showing various “countries” associated with the United States.</figcaption></figure><h4>Benefits and Limitations</h4><p>Getting data from APIs can vary in complexity and security requirements can be off-putting. The way data is requested can also vary, as can the format in which it returns. Documentation is produced by the provider themselves, which can also be a bottleneck worth considering.</p><p>It is worth mentioning at this point that there are various packages created by developers to interact with APIs in simplified manners — we’ll talk about third-party Python libraries in the next section!</p><p>With all of that being said, there is plenty of rich data available for you to pull that may be utilised in customer-facing data products as well as for demo purposes.</p><h3>Third-party packages</h3><p>There are also some great ready-made packages out there for us to use. This takes some of the heavy effort out of finding a source, processing and formatting the data ready for consumption.</p><p><a href="https://faker.readthedocs.io/en/master/"><em>Faker</em></a> is one such example, with the ability to generate names, addresses, etc.</p><p>Install, import and use packages like this at your convenience!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/34843ecf1a2ee6a08c0d818daad9f083/href">https://medium.com/media/34843ecf1a2ee6a08c0d818daad9f083/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AOxkv9PTN1By_w0McJzn6Q.png" /><figcaption>Image by Author: Screenshot of several Faker generated names</figcaption></figure><h4>Benefits and Limitations</h4><p>As much of the leg-work has been done for you by other developers, packages such as <em>Faker</em> can be very high-impact and low-cost (in both price and time).</p><p>With just <em>Faker</em> in mind, we’ve already been able to generate generic and informative user data intuitively. Other libraries are available (often interacting with common APIs) to allow developers to easily pull useful through to their applications.</p><p>Blockers could lie within your organisation’s regulations around using third-party packages, licensing and maintainability of the package itself, so please be aware of possible restrictions here.</p><h3>ChatGPT</h3><p>It wouldn’t be fitting to overlook Large Language Models (LLMs) as they are a great asset in data generation.</p><p>A simple approach could be to get ChatGPT (or equivalent) to generate you a list of data points, such as possible customer names.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fg9XlChnYr9i0_0A8VstHg.png" /><figcaption>Image by Author: Asking ChatGPT for a list of names.</figcaption></figure><p>We can do one better and ask for the LLM to help build the random variable function itself. You can include details about data sources and even ask ChatGPT to write a function to interact with an API you have found.</p><p>An example here is requesting help with the <a href="https://www.boredapi.com/">bored</a> API, extracting the activity field and returning it for your own use:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HmNUz-SC53YhuyJoKNqXdw.png" /><figcaption>Image by Author: Asking ChatGPT for a random activity.</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9915711c90082ab2fc988a863351a22e/href">https://medium.com/media/9915711c90082ab2fc988a863351a22e/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J0QLdorULTaoepdtJ9zLGA.png" /><figcaption>Image by Author: Example result of ChatGPT generated function from bored API.</figcaption></figure><p><strong><em>Note: Always check the output of your result to ensure an LLM generated function is working as expected.</em></strong></p><h4>Benefits and Limitations</h4><p>Generative AI is showing its strengths in rapid data generation (prominently around content creation, as shown by <a href="https://www.gartner.com/en/doc/779868-emerging-tech-top-use-cases-for-generative-ai">this Gartner report</a>) and I expect to see more grounded approaches for this in populating example datasets next year. As such, it makes creation of various data points at scale accessible to users of various backgrounds and levels of technicality.</p><p>With that being said, reliability and consistency of LLMs is being heavily investigated at the same time. Depending on the type of data you need to populate, this may introduce rigorous checks into data quality and anonymity and determine whether or not this is the right tool for specific use cases.</p><h3>Putting this into a DataFrame</h3><p>The approach we’ve taken across the previous sections is to generate our data ready to be processed into the DataFrame column of interest.</p><p>There are a few choices of how to implement the synthetic data at this stage. UDFs are a neat way to apply a function for each row of the column and can help define uniqueness.</p><p><a href="https://github.com/databrickslabs/dbldatagen">Databricks Labs Data Generator</a> is another great library that can help speed things up with distributed computing at the core of this data generation.</p><p>Manipulating how we generate the lists of values to be populated, we can implement the <em>withColumn</em> PySpark function and combine this with our list data to produce our text column at great speed.</p><p>For our test case, we’ll use a combination of both of these approaches. We can use Databricks Labs Data Generator to create our DataFrame wireframe, populating 100 rows with a unique <em>Id</em> column, and any values we might not require to be unique, such as <em>Country</em>.</p><p>Other columns which we might wish to be distinct, such as the <em>Full Name</em> and <em>Username</em>, we can make use of a UDF to use the id column to access a unique name from the list we’ve created.</p><p>Putting it all together, we can create our DataFrame:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/06ac5e39afb69d1ec9e744fe2e080db0/href">https://medium.com/media/06ac5e39afb69d1ec9e744fe2e080db0/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*spZTGH7jOIReaNpiFspCwQ.png" /><figcaption>Image by Author: DataFrame</figcaption></figure><p>A quick data profile shows that our <em>Full Name</em> and <em>Username</em> columns have completely unique values as desired!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y1DylIGYOTXIlfXFwxmgrQ.png" /><figcaption>Image by Author: Data profile of Categorical columns showing 100 unique values for “Full Name” and “Username” columns</figcaption></figure><h3>Conclusion</h3><p>We’ve outlined a variety of methods to generate textual synthetic data quickly, allowing us to accelerate our demonstrative dataset creation.</p><p>All the examples above can be extended, refined and tailored to your specific use-case.</p><p>Are there any tricks that I have missed? What are you using synthetic datasets for? Let me know in the comments!</p><p>Thanks for reading, and as always, feel free to find the code for your own use <a href="https://github.com/MattPCollins/SyntheticDimensionTableExamples">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c6678cc10aff" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/methods-for-generating-synthetic-descriptive-data-c6678cc10aff">Methods for generating synthetic descriptive data</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Create Many-To-One relationships Between Columns in a Synthetic Table with PySpark UDFs]]></title>
            <link>https://medium.com/data-science/create-many-to-one-relationships-between-columns-in-a-synthetic-table-with-pyspark-udfs-41e39d97c936?source=rss-d1970f1605f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/41e39d97c936</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-modeling]]></category>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Matt Collins]]></dc:creator>
            <pubDate>Sat, 09 Dec 2023 07:14:13 GMT</pubDate>
            <atom:updated>2023-12-09T07:14:13.144Z</atom:updated>
            <content:encoded><![CDATA[<h4>Leverage some simple equations to generate related columns in test tables.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8-dsn4Kccx_xN6of_HzE6A.jpeg" /><figcaption>Image generated with DALL-E 3</figcaption></figure><p>I’ve recently been playing around with Databricks Labs Data Generator to create completely synthetic datasets from scratch. As part of this, I’ve looked at building sales data around different stores, employees, and customers. As such, I wanted to create relationships between the columns I was artificially populating — such as mapping employees and customers to a certain store.</p><p>Through using PySpark UDFs and a bit of logic, we can generate related columns which follow a many-to-one relationship. With a bit of magic, we are even able to extend the logic to give some variance in this mapping — like a customer generally buying from their local store but sometimes from a different store.</p><h3><strong>Use Databricks Labs Data Generator to generate our basic DataFrame</strong></h3><p><strong><em>Note: You can skip this section if not required!</em></strong></p><p>First up, we need to create a DataFrame with our first randomly-generated column. In our case, we’re going to start with the store, as logically we will have “many employees per store” and “many customers repeatedly shopping at a store”.</p><p>With a Star Schema Data Model in mind, we’re going to start with our Sales Fact table — a transactional table which will contain key values for the Sale Id, Store Id, Employee Id and Customer Id, the Sale Amount along with some datetime data for the purchase. We can then fill out the specifics about the Store, Employee and Customer in dimension tables further down the line.</p><p>We’ll start small — a table with 1000 sales will do. We now need to decide how to split these sales up between stores, employees and customers. Let’s suggest the following:</p><ul><li># Stores = 20</li><li># Employees = 100</li><li># Customers = 700</li></ul><p>We can also say that the sales will be recorded over the course of last month:</p><ul><li>First Sale Date = 2023–11–01</li><li>Last Sale Date = 2023–11–30</li></ul><p>The Sale Id needs to be a unique column so we can generate an Id column for this. We now need to distribute the 1000 sales across the 20 stores. For simplicity we will assume this is random.</p><p>Using Databricks Lab Generator we can do this with the following code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1dbbf19e973a3afea5526d7c613ffc8d/href">https://medium.com/media/1dbbf19e973a3afea5526d7c613ffc8d/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b66c9f36f7f8e0378e89d2839eab7dfa/href">https://medium.com/media/b66c9f36f7f8e0378e89d2839eab7dfa/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/87284680b5d1453a517dc9ba491d5169/href">https://medium.com/media/87284680b5d1453a517dc9ba491d5169/href</a></iframe><p>Now add some code to record when the sales were made and their amount. To keep things simple, we’ll round the timestamp of the sale to the nearest hour.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7a068c3c2af10db25a149bc117aa9831/href">https://medium.com/media/7a068c3c2af10db25a149bc117aa9831/href</a></iframe><p>To calculate the sale amount, we can use the “expr” parameter in our withColumn expression to allow us to generate a random number, with some rules/boundaries.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/527be3b6e46203b82d69b71f3e118c05/href">https://medium.com/media/527be3b6e46203b82d69b71f3e118c05/href</a></iframe><p>In this case, the expression is quite straight-forward: produce a random number (between 0 and 1), add 0.1 (ensuring sale values are not 0) and multiply by 350.</p><p>We’ve got our basic shape for the DataFrame now, so put it all together:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0dcda45fd358954946a751d2277a66d3/href">https://medium.com/media/0dcda45fd358954946a751d2277a66d3/href</a></iframe><p>We can create a quick Data Profile to look at the distribution of values in the columns:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CLxlYlD85N_6PnxESK06Xg.png" /><figcaption>Image by Author: Data profile generated in Databricks</figcaption></figure><p>We can see that the StoreId distribution is relatively even across the 20 stores, with no missing values and averages around the centre as we would expect. The same follows for the Timestamps and amount values.</p><h3><strong>Add a dependent Many-To-One column</strong></h3><p>Now we can add our Employee Id column to the DataFrame. We’re done with Databricks Lab Data Generator now, so will just use PySpark operations to add columns to the DataFrame.</p><p>Stepping back from the code, we want to model this as the following statements:</p><ul><li>There are 20 stores.</li><li>Each store has more than 1 employee.</li><li>Each employee works at a single store only.</li></ul><p>First we need to split the employees between the stores. The following python function can be used to do so:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ccc1120174ff30fa5f4b3fbc78243d85/href">https://medium.com/media/ccc1120174ff30fa5f4b3fbc78243d85/href</a></iframe><p>Now that we have our distribution of employees for each store, let’s start assigning Ids!</p><p>The employeesPerStore list ensures that the employee Ids per store do not overlap. We can use this to randomly assign an employee Id to a sale in the table with the following equation:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/de106d99c7675d2dcadbaa794a484ca3/href">https://medium.com/media/de106d99c7675d2dcadbaa794a484ca3/href</a></iframe><p>This function currently only works for a single value — we need to put this into something that a PySpark DataFrame can work with (functionally, and quickly!)</p><p>We can pass PySpark UDFs to the <em>withColumn</em> method, so let’s reformat this logic into a function and set it to a UDF:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5061f4dc681c9a78a07158aa452cd14d/href">https://medium.com/media/5061f4dc681c9a78a07158aa452cd14d/href</a></iframe><p>Now call this as a new column in the DataFrame:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c0ab8deeb651ff670c54ce53e136881e/href">https://medium.com/media/c0ab8deeb651ff670c54ce53e136881e/href</a></iframe><p>We can quickly test this looks correct by using the Visualisation tool in Databricks to see the distinct count of Employee Ids per Store Id. This is my preference but you could also use group by logic or other plotting modules, if desired.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*adqA3ITRwEmFjs-O3NzZxQ.png" /><figcaption>Image by Author: Distinct count of Employee Ids per Store</figcaption></figure><p><strong>Important Note: </strong>This logic allows for employees to be<strong> missed </strong>from the results. This means that it is possible for an employee to make 0 sales, and thus not be included in the DataFrame. We’ll look at how to ensure all customers have sales recorded against them in the next section.</p><h3>Add the customers column</h3><p>The customers column is a bit different… while our use-case suggests it is common for a customer to shop at a single store multiple times, it is entirely possible that they got to a different store one day. How do we model this?</p><p>We’ve got the starting points with the work done for our employees column, so can repeat the <em>get_employees </em>function and UDF logic for customers as below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/38ba0fef63c1b950c9dc9ffe815daa94/href">https://medium.com/media/38ba0fef63c1b950c9dc9ffe815daa94/href</a></iframe><p>We’ve again potentially missed a few customers off here. Here are a few approaches to rectify this:</p><ul><li>Recalculate in <em>while</em> loop until you converge on a DataFrame which contains all customers (inefficient, costly, could run indefinitely)</li><li>Randomly update customer Ids in <em>while </em>loop until all customers in DataFrame (requires logic to only overwrite same stores, could also run indefinitely)</li><li>Return a list of all customer Ids with more than 1 record in the sales table, and randomly overwrite until all missing Ids are added (also needs logic for overwriting customers in same store, may also require <em>while</em> loop logic)</li><li><strong>Reverse the process and start with employees. This ensures each employee is randomly assigned to rows. We can then use the mapping and apply the store Id.</strong></li></ul><p>Hopefully it is clear why the last option is the lowest effort to compute — we have all the code required so just need to reformat things slightly.</p><p>Our new scripts looks as follows:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0637493d35285e5e960557956c07ab6e/href">https://medium.com/media/0637493d35285e5e960557956c07ab6e/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1fb4045e2ea95a78910da70d0ee5dc0f/href">https://medium.com/media/1fb4045e2ea95a78910da70d0ee5dc0f/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qV_RJv9IJsS2cPQAFF3fkA.png" /><figcaption>Image by Author: Databricks Data Profile for the new DataFrame</figcaption></figure><h3>Adding Randomness to the customers</h3><p>What we now need is a bit of randomness, which we need to define. For our example, let’s say that each customer has a 90% chance of shopping at the usual store (the “local” store). If we do not need all customers to be returned in the results set, we can simply adjust our <em>customers_udf</em> as follows, and use <em>df2</em>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3bfdda7d48a67575403095322d4782eb/href">https://medium.com/media/3bfdda7d48a67575403095322d4782eb/href</a></iframe><p>The logic involves using the <em>random.choices </em>function to supply a weighted list and return a single value.</p><p>To compute the weighted list, we have the weight of our “local” store for the customer, in this case 90%, so need to assign the remaining 10% to the other stores, in this case 19 stores. The probability of each other store being selected will therefore be 10/19 = 0.526%. We can populate an array with these percentages, which would look something like the following:[0.526,0.526,0.526,…,90,0.526,…0.526]</p><p>Passing this into random.choices, we then randomly select a store Id from the list with the corresponding weights and use this as the input for the <em>customer_id </em>variable, as before.</p><p><strong>Note:</strong> The output of random.choices returns a list (as you can request k results), so access the 0th element of the list to get the store_id as an integer value.</p><p>If we need to combine this logic with a DataFrame including all customers, we can reverse the process slightly. The weights logic is still valid so we can just plug this into randomly select a store and return this as the result:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ead16819022b7f36be7f734b577bf9fa/href">https://medium.com/media/ead16819022b7f36be7f734b577bf9fa/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mw8sT84v_DGi9vo8zD9bSA.png" /><figcaption>Image by Author: Sample of final DataFrame in Databricks</figcaption></figure><h3>Conclusion</h3><p>There we have it! A synthetically created DataFrame with both strict and loose mappings between columns. You can now progress the next steps to populate related tables which may contain more descriptive information, such as dimension tables of store names, addresses, employee names, roles, etc. This can also be done using Databricks Labs Data Generator or any other tool/process you are comfortable with.</p><p>There are some great examples on the Databricks Labs Data Generator <a href="https://github.com/databrickslabs/dbldatagen">GitHub Repo</a> along with documentation, so please do take a look at this if you are curious to learn more.</p><p>All of my code can be accessed from the following <a href="https://github.com/MattPCollins/Dbldatagen-Related-Tables">GitHub Repo</a>.</p><p>If you have any thoughts, comments or alternatives to this demo, please reach out in the comments. Thanks!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=41e39d97c936" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/create-many-to-one-relationships-between-columns-in-a-synthetic-table-with-pyspark-udfs-41e39d97c936">Create Many-To-One relationships Between Columns in a Synthetic Table with PySpark UDFs</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Parallelising Python on Spark: Options for concurrency with Pandas]]></title>
            <link>https://medium.com/data-science/parallelising-python-on-spark-options-for-concurrency-with-pandas-7ca553b9f265?source=rss-d1970f1605f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/7ca553b9f265</guid>
            <category><![CDATA[parallel-computing]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Matt Collins]]></dc:creator>
            <pubDate>Sat, 18 Nov 2023 05:04:31 GMT</pubDate>
            <atom:updated>2023-11-19T03:28:59.012Z</atom:updated>
            <content:encoded><![CDATA[<h3>Parallelising Python on Spark: Options for Concurrency with Pandas</h3><h4>Leverage the benefits of Spark when working with Pandas</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fC2PBZiShJmAYu0T" /><figcaption>Photo by <a href="https://unsplash.com/@flo_stk?utm_source=medium&amp;utm_medium=referral">Florian Steciuk</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In my previous role, I spent some time working on an internal project to predict future disk storage space usage for our Managed Services customers across thousands of disks. Each disk is subject to its own usage patterns and this means we need a separate machine learning model for each disk which takes historical data to predict future usage on a disk-by-disk basis. While performing this prediction and choosing the correct algorithm for the job is a challenge in itself, performing this at scale has its own problems.</p><p>In order to take advantage of more sophisticated infrastructure, we can look to move away from sequential predictions and speed up the operation of the forecasting by parallelising the workload. This blog post aims to compare Pandas UDFs and the ‘concurrent.futures’ module, two approaches of concurrent processing, and determine use cases for each.</p><h3>The Challenge</h3><p>Pandas is a gateway package in Python for working with datasets in the analytics space. Through working with DataFrames, we’re able to profile data and evaluate data quality, perform exploratory data analysis, build descriptive visualisations of the data and predict future trends.</p><p>While this is certainly a great tool, the single-threaded nature of Python means it can scale poorly when working with larger data sets, or when you need to perform the same analysis across multiple subsets of data.</p><p>In the world of big data, we expect a bit more sophistication in our approach, as we have the additional focus on scalability to keep great performance. Spark, amongst other languages, allows us to take advantage of distributed processing to help us process larger and more complicated data structures.</p><p>Before digging into this specific example, we can generalise some use cases which summarise the need for concurrency in data processing:</p><ul><li>Apply uniform transformations to multiple data files</li><li>Forecast future values for several subsets of data</li><li>Tune hyperparameters for machine learning model and select most efficient configuration</li></ul><p>When escalating our requirement to perform workloads like those suggested above and in our case, the most straightforward approach in Python and Pandas is to process this data sequentially. For our example, we would run the above flow for one disk at a time.</p><h3>The Data</h3><p>In our example, we have data for thousands of disks that show the free space recorded over time and we want to predict future free space values for each of the disks.</p><p>To paint the picture a bit more clearly, I’ve provided a csv file containing 1,000 disks each with one month of historical data for free space measured in GB. This is of sufficient size for us to see the impact of the different approaches to predicting at scale.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/71690a9e472dfe4b288c79f6f3ae1b34/href">https://medium.com/media/71690a9e472dfe4b288c79f6f3ae1b34/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/310/0*ZGOlThNdZCkXzOhy.png" /><figcaption>Image by Author: Example DataFrame</figcaption></figure><p>For a time-series problem like this, we’re looking to use historical data to predict future trends and we want to understand which Machine Learning (ML) algorithm is going to be most appropriate for each disk. Tools like AutoML are great for this when looking to determine the appropriate model for one dataset, but we’re dealing with 1,000 datasets here — so this is excessive for our comparison.</p><p>In this case, we’ll limit the number of algorithms we want to compare to two and see which is the most appropriate model to use, for each disk, using the Root Mean Squared Error (RMSE) as a validation metric. Further information on RMSE can be found<a href="https://www.statology.org/how-to-interpret-rmse/"> here </a>. These algorithms are:</p><ul><li>Linear regression</li><li>Fbprophet (fitting the data to a more complex line)</li><li>Facebook’s time-series forecasting model.</li><li>Built for more complex predictions with hyperparameters for seasonality.</li></ul><p>We’ve got all the components ready now if we wanted to predict a single disk’s future free space. The set of actions follows the below flow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gJCTji1BXUPUgR7U.png" /><figcaption>Image by Author: Data Lifecycle</figcaption></figure><p>We now want to scale this out, performing this flow for multiple disks, 1,000 in our example.</p><p>As part of our review, we’ll compare the performance of calculating RMSE values for the different algorithms at different scales. As such, I’ve created a subset of the first 100 disks to mimic this.</p><p>This should give some interesting insights into performance on different-sized datasets, performing operations of varying complexity.</p><h3>Introducing concurrency</h3><p>Python is famously single-threaded and subsequently does not make use of all the compute resources available at a point in time.</p><p>As a result, I saw three options:</p><ol><li>Implement a for loop to calculate the predictions sequentially, taking the single-threaded approach.</li><li>Use Python’s <a href="https://docs.python.org/3/library/concurrent.futures.html">futures</a> module to run multiple processes at once.</li><li>Use Pandas UDFs (user-defined functions) to leverage distributed computing in PySpark while maintaining our Pandas syntax and compatible packages.</li></ol><p>I wanted to do a fairly in-depth comparison under different environment conditions, so have used a single-node Databricks cluster and another Databricks cluster with 4 worker nodes to leverage Spark for our Pandas UDF approach.</p><p>We’ll follow the following approach to evaluate the suitability of the Linear Regression and fbprophet models for each disk:</p><ul><li>Split the data into train and test sets</li><li>Use the training set as input and predict over the test set dates</li><li>Compare the predicted values with the actual values in the test set to get an Root Mean Squared Error (RMSE) score</li></ul><p>We’re going to return two things in our outputs: a modified DataFrame with the predictions, giving us the additional benefit of plotting and comparing the predicted vs actual values, and a DataFrame containing the RMSE scores for each disk and algorithm.</p><p>The functions to do so look like the below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/814fd111a43304a9067cb06d57325b03/href">https://medium.com/media/814fd111a43304a9067cb06d57325b03/href</a></iframe><p>We’re going to compare the three approaches outlined above. We’ve got a few different scenarios, so we can fill out a table of what we’re collecting results against:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1ce64c89b8c66c854bb827330b723220/href">https://medium.com/media/1ce64c89b8c66c854bb827330b723220/href</a></iframe><p>With the following combinations:</p><h4>Method</h4><ul><li>Sequential</li><li>futures</li><li>Pandas UDFs</li></ul><h4>Algorithm</h4><ul><li>Linear regression</li><li>Fbprophet</li><li>Combined (both algorithms for each disk) — most efficient way to gather a comparison.</li></ul><h4>Cluster mode</h4><ul><li>Single Node Cluster</li><li>Standard Cluster with 4 workers</li></ul><h4>Number of disks</h4><ul><li>100</li><li>1000</li></ul><p>The results are presented in this format in the appendix of this blog, should you wish to take a further look.</p><h3>The Methods</h3><h4>Method 1: Sequential</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/77cf2a86eb30db4af2bbb074b147b7d0/href">https://medium.com/media/77cf2a86eb30db4af2bbb074b147b7d0/href</a></iframe><h4>Method 2: concurrent.futures</h4><p>There are two options in using this module: parallelising memory-intensive operations (using ThreadPoolExecutor) or CPU-intensive operations (ProcessPoolExecutor). One descriptive explanation of this is found in the following <a href="https://medium.com/@bfortuner/python-multithreading-vs-multiprocessing-73072ce5600b">blog</a>. As we’re going to be working on a CPU intensive problem, ProcessPoolExecutor is fitting for what we’re trying to achieve.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b9e4e5c55d629647d84c6cc0647663dc/href">https://medium.com/media/b9e4e5c55d629647d84c6cc0647663dc/href</a></iframe><h4>Method 3: Pandas UDFs</h4><p>Now we’re going to switch gear and use Spark and leverage distributed computing to help with our efficiency. Since we’re using Databricks, most of our Spark configuration is done for us but there are some tweaks to our general handling of the data.</p><p>First, import the data to a PySpark DataFrame:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b1250b1d38b8f200acebef6e1e8767a9/href">https://medium.com/media/b1250b1d38b8f200acebef6e1e8767a9/href</a></iframe><p>We’re going to make use of the Pandas grouped map UDF (PandasUDFType.GROUPED_MAP), since we want to pass in a DataFrame and return a DataFrame. Since <a href="https://databricks.com/blog/2020/05/20/new-pandas-udfs-and-python-type-hints-in-the-upcoming-release-of-apache-spark-3-0.html">Apache Spark 3.0</a> we don’t need to explicitly declare this decorator anymore!</p><p>We need to split out our fbprophet, regression and RMSE functions for Pandas UDFs due to DataFrame structuring in PySpark, but don’t require a massive code overhaul to achieve this.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9d9698541bbcaf12fd8ae0974536c05a/href">https://medium.com/media/9d9698541bbcaf12fd8ae0974536c05a/href</a></iframe><p>We can then use applyInPandas to produce our results.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9a3bb804e89eb57a04479110e0d7591e/href">https://medium.com/media/9a3bb804e89eb57a04479110e0d7591e/href</a></iframe><p>Note: the examples above are only demonstrating the process for using Linear Regression for readability. Please see the full <a href="https://github.com/MattPCollins/ConcurrentModelTraining/blob/main/concurrent_model_training.ipynb">notebook</a> for the complete demonstration of this.</p><h3>Interpreting the results</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wtIOu5EoRJzpAnwY.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vQg6Lx-34ZTVRqqM.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eda2K998rPfAeNZp.png" /><figcaption>Images by Author: Execution durations for forecasting algorithms, for each suggested method</figcaption></figure><p>We’ve created plots for the different methods and different environment set-ups, then grouped the data by algorithm and number of disks for easy comparison.</p><p>Please note that the tabular results are found in the appendix of this post.</p><p>I’ve summarised the highlights of these findings below:</p><ul><li>As expected, predicting 1,000 disks compared to 100 disks is (generally) a more time-consuming process.</li><li>The sequential approach is generally the slowest, being unable to take advantage of underlying resources in an efficient manner.</li><li>Pandas UDFs are quite inefficient on the smaller, simpler tasks. The overhead of transforming the data is more expensive — parallelising helps to compensate for this.</li><li>Both sequential and concurrent.futures approaches are oblivious to the clustering available in Databricks — missing out on additional compute.</li></ul><h3>Closing thoughts</h3><p>Context certainly plays a big part in which approach is most successful, but given Databricks and Spark are often used for Big Data problems, we can see the benefit of using Pandas UDFs with those larger, more complex datasets that we’ve seen here today.</p><p>Using a Spark environment for smaller datasets can be done just as efficiently on a smaller (and less expensive!) compute configuration at great efficiency as demonstrated by the use of the concurrent.futures module, so do bear this in mind when architecting your solution.</p><p>If you’re familiar with Python a­nd Pandas then neither approach should be a strenuous learning curve to move away from the sequential for loop approach seen in beginner tutorials.</p><p>We’ve not investigated it in this post as I have found discrepancies and incompatibilities with the current version, but the recent pyspark.pandas module will certainly be more common in the future, and one approach to look out for. This API (along with Koalas, developed by the guys at Databricks, but now retired) leverages the familiarity of Pandas with the underlying benefits of Spark.</p><p>For demonstrating the effect we are trying to achieve, we’ve only gone as far as to look at the RMSE values produced for each disk, rather than actually predict a future time-series set of values. The framework we’ve set up here can be applied in the same way for this, with logic to determine if the evaluation metric (along with other logic, such as physical limitations of a disk) is appropriate in each case and to predict the future values, where possible, using the determined algorithm.</p><p>As always, the notebook can be found in my<a href="https://github.com/MattPCollins/ConcurrentModelTraining"> GitHub </a>.</p><h3>Appendix</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ca8f9ab6d05ae4b3f83adf3ee1a288a0/href">https://medium.com/media/ca8f9ab6d05ae4b3f83adf3ee1a288a0/href</a></iframe><p><em>Originally published at </em><a href="https://blog.coeo.com/parallelising-python-on-spark-different-approaches"><em>https://blog.coeo.com</em></a>, adapted for this repost<em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7ca553b9f265" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/parallelising-python-on-spark-options-for-concurrency-with-pandas-7ca553b9f265">Parallelising Python on Spark: Options for concurrency with Pandas</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Three ways to profile data with Azure Databricks]]></title>
            <link>https://medium.com/@mc12338/three-ways-to-profile-data-with-azure-databricks-5d8e816f670f?source=rss-d1970f1605f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/5d8e816f670f</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-profiling]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Matt Collins]]></dc:creator>
            <pubDate>Thu, 16 Nov 2023 04:42:26 GMT</pubDate>
            <atom:updated>2023-11-16T04:42:26.242Z</atom:updated>
            <content:encoded><![CDATA[<h3>Three Ways to Profile Data with Azure Databricks</h3><h4>Get a feel for your data quality and shape quickly with data profiling</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CwDkabQ4zvK7LEen" /><figcaption>Photo by <a href="https://unsplash.com/@chrisliverani?utm_source=medium&amp;utm_medium=referral">Chris Liverani</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Data quality is an increasingly important part of generating successful and meaningful insights for data-driven businesses. At a basic level, missing data can lead to an incomplete picture of the business situation, leading to poor decisions and stakeholders asking “what did we miss?”.</p><p>This blog aims to highlight a few of the options available to us in Azure Databricks to profile the data and understand any data cleansing or feature engineering tasks required before we dive in and produce the consumable reports and cutting-edge forecasting which deliver the ultimate business/user value.</p><h3>Motivation</h3><p>The purpose of investigating the quality of the data is to understand what we are pulling through and ensure it is formatted and organised appropriately, allowing us to make better decisions. There are a few things we typically look at during this stage, outlined below:</p><h4>Data quality</h4><ul><li>Missing data</li><li>Duplicate data</li><li>Data types (this can often come in different shapes and be written in clashing formats, especially dates)</li><li>Inconsistent formatting</li></ul><p>We’d like to highlight and standardise these things during our Data Engineering and ETL processes.</p><h4>Shape of data</h4><ul><li>Row count</li><li>Uniqueness</li><li>Distribution of values (are we skewed to 90% of records having the same value, does it follow a uniform distribution?)</li></ul><p>As part of the above tasks, we’ll also demonstrate our domain knowledge through these questions:</p><p><em>Why is this data relevant to stakeholders?</em></p><ul><li>This is probably captured in the requirements gathering stage when the data project kicks off but is an excellent refresher to understand if missing data being pulled back from the source impacts the stakeholder in any way.</li></ul><p><em>What are the feature engineering requirements?</em></p><ul><li>The data we gathered may not capture the complete pictures we are trying to paint and some adjustments need to be made. This could be aggregations applied over the wrong timespan, normalisation to prevent features from having dominant weights in ML computations, or categorical encoding to include categorical data in classification tasks.</li><li>The aim is interpretability and data profiling is the perfect place to discover just how “user-friendly” our columns are.</li></ul><h3>Notebook data profile</h3><p>The easiest way to get started is to return your dataset as a DataFrame in a language of your choice (Python/Pandas/PySpark, Scala, SQL, r). I’ve imported a CSV of <a href="https://www.kaggle.com/datasets/jonbown/simpsons-episodes-2016/code?select=simpsons_characters.csv">Simpsons Episode data</a> from Kaggle.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7f8084b44a930e77317ca1c715ed7210/href">https://medium.com/media/7f8084b44a930e77317ca1c715ed7210/href</a></iframe><p>We can see the tabular results using the display function, but there is one additional option, to select the data profile.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-BmWBAnBb6K3ryrq.png" /><figcaption>Image by Author: Databricks data profile</figcaption></figure><p>Here we get the descriptions you would see in functions like df.describe() along with a histogram showing the distribution of values. Alongside quickly picking up on missing values, we’re able to quickly spot quirks of the data such as unbalanced distributions or unexpected max values, possibly the result of user error.</p><h3>AutoML</h3><p><a href="https://www.databricks.com/product/automl">AutoML</a> is a tool that allows users to quickly generate machine learning models for a specified dataset at the click of a button. By specifying a column you wish to predict or classify, AutoML prepares the dataset for training and runs various trials and their evaluation metrics for you to determine appropriate machine learning models to investigate further with reproducible notebooks and artefacts for comparison.</p><p>AutoML isn’t just great for jumpstarting your Machine Learning modelling process though. As part of gaining some understanding of the data, it runs the <a href="https://github.com/ydataai/pandas-profiling">pandas_profiling </a>module to produce a data exploration notebook, which you can open and see similar graphical explorations of your dataset seen in the Databricks data profiling tool. The data exploration notebook contains the following breakdown of the data:</p><p><em>Overview, alerts and reproduction</em></p><ul><li>The overview provides high-level summaries of the dataset looking at counts of variables, observations, missing data details and duplicates.</li><li>Alerts inform us of the columns where we may have incomplete or redundant data.</li><li>Reproduction gives info on config options along with how long the profile took to generate.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7fG6bEEyENTQkkzN.png" /><figcaption>Image by Author: Pandas Profiling overview statistics</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SUhZiCfSZRFrL9_a.png" /><figcaption>Image by Author: Pandas Profiling alerts statistics</figcaption></figure><p><em>Variables</em></p><ul><li>Here we get the visual information on the spread of data alongside any warnings flagged up in the alerts. This is a bit more verbose than the information returned in the default data profile.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fj8S_5vZNhvGc8gc.png" /><figcaption>Image by Author: Pandas Profiling Variable statistics</figcaption></figure><p><em>Interactions</em></p><ul><li>Scatter plot of numerical data columns</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*38uaJ0qBfjRwtDaO.png" /><figcaption>Image by Author: Pandas Profiling scatter plot</figcaption></figure><p><em>Correlations</em></p><ul><li>Shows how tightly correlated numerical points are</li><li>Different correlation coefficients available</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*a-pUbL90qa9YDTbN.png" /><figcaption>Image by Author: Pandas Profiling correlation plot</figcaption></figure><p><em>Missing values</em></p><ul><li>There are a few views to understand your missing data a bit better</li></ul><p><em>Data sample</em></p><ul><li>For those who enjoy a tabular view of the data</li></ul><p>To use this great tool, you must save your DataFrame as a Hive table, which is as straight-forward as the following command:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/42402b7b1b96530a13039cb527c8e0ed/href">https://medium.com/media/42402b7b1b96530a13039cb527c8e0ed/href</a></iframe><p>Instructions on running an AutoML experiment to generate this Profile notebook can be found <a href="https://docs.microsoft.com/en-us/azure/databricks/applications/machine-learning/experiments-page">here.</a></p><h3>Custom queries and visualisations</h3><p>We can naturally get all of the above information ourselves by querying the data directly in whichever is our preferred language in Databricks, along with further insights we may have an interest in.</p><p>With domain knowledge and understanding of the data, we know the questions stakeholders wish to discover the answers to. Targeted analysis at this stage is important in scoping whether you have sufficient information to answer their questions. If not, is feature engineering required or is additional web-scraping/data collection needed to fill these blanks? This is something a generic profiling tool can’t do, but they do allow us to take some of the legwork out so we can focus on the questions specific to our business needs.</p><p>Databricks’ easy-to-use Data Visualisations based on our queries, paired with using a variety of languages, provides the ability to perform data exploration before the need to build a clean data model.</p><p>In this example, we might want to investigate the importance of a character in the episode title:</p><p>Such as the Average rating of episodes over time:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_DVpUZOobGSPMYRg.png" /><figcaption>Image by Author: Line chart example</figcaption></figure><p>Or the consistency of episode ratings for the main characters:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lQw_rXkMNmkbdyAa.png" /><figcaption>Image by Author: Box plot example</figcaption></figure><p>Another line of investigation might be to look at the recurring words in the titles, and if this is a predictor of success or viewership.</p><p>Word clouds are a selectable visualisation in Databricks (no need to download a python module!)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/322/0*0DcakqmLPNoLQWZD.png" /><figcaption>Image by Author: Word cloud example</figcaption></figure><p>This can in turn give some quick insights into the success of these commonly used words (once we have filtered out the less informative filler words such as “the” and “and”).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YtkPw5_MEw2w5Yqd.png" /><figcaption>Image by Author: Scatter plot example</figcaption></figure><p>For example, we see consistency in the success of the Treehouse of Horrors Halloween specials in each season — are there any other patterns worth reviewing?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZU_5DjZpIYZe6g0M.png" /><figcaption>Image by Author: Baseline comparison to Halloween special episodes</figcaption></figure><p>This might provide an incentive to pull transcript data from the episodes for analysis of dialogue, or word counts per character for themes or character pairings that are successful.</p><p>The notebook with the source code to produce these visuals can be found in my GitHub repo linked <a href="https://github.com/MattPCollins/DatabricksDataProfiling">here</a>.</p><h3>Comparison</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FyTx4KETvd0-GO5W53bNPA.png" /><figcaption>Image by Author: Advantages and disadvantages of each approach</figcaption></figure><h3>Closing Thoughts</h3><p>Identifying data quality issues early in your data lifecycle is beneficial to efficiently move data from raw to consumable format. The visual inspection of your data helps to quickly identify gaps that may be required to build the complete picture which is required for analysis and reporting further down the line.</p><p>That being said, this process can be beneficial to multiple teams in your data department. Data Engineers can review the quality of the raw data being ingested and Data Scientists can start to understand dimensionality reduction opportunities with the transformed level.</p><p><em>Originally published at </em><a href="https://blog.coeo.com/three-ways-to-profile-data-with-azure-databricks"><em>https://blog.coeo.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5d8e816f670f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Classification Model Serving bug on Databricks cluster runtime 12.2 LTS ML]]></title>
            <link>https://medium.com/@mc12338/classification-model-serving-bug-on-databricks-cluster-runtime-12-2-lts-ml-dd24692a598e?source=rss-d1970f1605f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/dd24692a598e</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[mlflow]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Matt Collins]]></dc:creator>
            <pubDate>Mon, 15 May 2023 10:09:23 GMT</pubDate>
            <atom:updated>2023-06-19T14:56:01.837Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XIpEzNYwQM3B7euh" /><figcaption>Photo by <a href="https://unsplash.com/@lazycreekimages?utm_source=medium&amp;utm_medium=referral">Michael Dziedzic</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><em>Updated (19/06/23): Databricks have identified this model serving error and provided a notebook to add the pandas dependency in the following </em><a href="https://docs.databricks.com/machine-learning/automl/train-ml-model-automl-api.html#no-module-named-pandascoreindexesnumeric"><em>documentation</em></a><em>.</em></p><p>I’ve recently had some struggles with creating serving endpoints for various classification models in the MLflow model registry. I noticed the following errors for various Classification models created with AutoML experiments in my Databricks workspace when using a cluster with runtime 12.2 LTS ML.</p><p><em>An error occurred while loading the model. No module named ‘pandas.core.indexes.numeric’.</em></p><p>In this post, we’ll take a look at how to recreate the issue and why it seems to occur.</p><h4>TL;DR</h4><p>If you see this issue, the quickest fix to upgrade the pandas dependency as per the <a href="https://docs.databricks.com/machine-learning/automl/train-ml-model-automl-api.html#no-module-named-pandascoreindexesnumeric">Databricks documentation</a>. Alternatively, using a 11.3 LTS ML cluster is still a valid approach. For both solutions, re-run your experiment, re-deploy your model and serving endpoint and this should be successful! Read on for further information and workarounds.</p><h3>Environment Configuration</h3><p>For this, I’m using a Databricks workspace in Azure. The details are as follows:</p><ul><li>Resource Location: West Europe</li><li>Pricing Tier: Premium</li><li>Compute Runtime: 12.2 LTS ML</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wJYQGUfs6wTkkXjDa5nTrA.png" /><figcaption>Compute configuration: Image by author</figcaption></figure><h3>A Note On Serving Endpoints</h3><p>The resources associated with the endpoint are fully managed by Databricks and do not appear in your cloud account, abstracting administrative responsibilities from users. While this has its benefits, it does complicate and limit troubleshooting abilities in case of error (such as the one seen here).</p><p>For those who would like more control of their model endpoints in the cloud, MLflow can be integrated for inference via Docker containers or Azure Machine Learning solutions. More information is available <a href="https://learn.microsoft.com/en-us/azure/databricks/mlflow/log-model-dependencies#online-serving-third-party-systems-or-docker-containers">here</a>.</p><h3>Classification Example</h3><p>I encountered this error for a variety of Classification models, including a demo provided by Databricks themselves, so let&#39;s use this as our example.</p><p>The script linked <a href="https://docs.databricks.com/_extras/notebooks/source/machine-learning/automl-classification-example.html">here</a> performs classification on census income data. The script triggers an AutoML experiment (with a timeout set to 30 minutes) that generates various experiment runs which look to classify if a record will have an income of ≥ $50k &lt; $50k.</p><p>Running these commands in a notebook connected to this cluster will complete successfully we’ll see all runs associated with this AutoML experiment.</p><p>Choose the best one and register the best model to the Model Registry, you can do this through the GUI by navigating to the desired experiment run, and hit “Register Model”. You’ll be given the choice to create a new model if you haven’t already.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FvvssktLxO2Zn0WQW8-Y6Q.png" /><figcaption>Registered Model from our experiment: Image by author</figcaption></figure><p>From here it is straightforward to create a serving endpoint — which is intended to allow users to query the model via a REST API in real-time. Behind the scenes, Databricks sets up a container and deploys the model using the model artifacts saved as part of the experiment run.</p><p>We can see these details by looking at the experiment run itself. Notably, we’ll see a conda yaml file which contains details on the environment dependencies and module versions alongside a pickled model (Model.pkl) and requirements file.</p><p>This is where we start having issues. Set up the serving endpoint either from the Registered Model via the “Use model for inference” button, or via the “Serving” shortcut in the Machine Learning persona pane:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EyY6iuBQx488fQ_EiWBjXA.png" /><figcaption>Serving Endpoint Creation GUI: Image by author</figcaption></figure><h3>The Error</h3><p>The model spends a few minutes deploying and this is where we see our first error in the “State” field. Navigating to the logs, we can see some more information:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_MnrBcbYtgiWDxqVO2AQ-A.png" /><figcaption>Several error messages reported while deploying the endpoint in the service logs: Image by author</figcaption></figure><p>Looking back to our conda yaml and requirements files, we can sort of see why this might have occurred — there is no mention of the pandas module in the dependencies. So how do we fix this?</p><h3>The Easy Answer</h3><p>Switching your cluster runtime to 11.3 LTS ML is an instant fix for this — rerun your experimentation, register your best run to the registry and then update your serving endpoint to the new version and you should see this complete successfully.</p><p>While it is not completely obvious why the 12.2 runtime does not work, checking the <a href="https://learn.microsoft.com/en-gb/azure/databricks/release-notes/runtime/releases">Documentation</a> on the Databricks runtime releases shows a few differences which may explain this:</p><p>Runtime version: 12.1–12.2 LTS <br>MLflow version: <a href="https://www.mlflow.org/docs/2.1.1/index.html">2.1.1</a></p><p>Runtime version: 11.3 LTS<br>MLflow version: <a href="https://www.mlflow.org/docs/1.29.0/index.html">1.29.0</a></p><p>This is a fairly major difference in the MLflow version, which may impact the behaviour of serving endpoints. There are also slight differences in pandas versions between the runtimes:</p><p>Runtime version: 12.1–12.2 LTS<br>Pandas version: 1.4.2</p><p>Runtime version: 11.3 LTS<br>Pandas version: 1.3.4</p><h3>An Alternative Fix</h3><p>Downgrading compute runtimes isn’t an option for everyone, so there are thankfully a few tricks we can do, one is setting the conda environment dependencies ourselves manually.</p><p>Get the existing conda environment logged to the experiment you tried to set up the endpoint for. Navigate to your registered model &gt; select the version &gt; click the link to the source run &gt; click on the conda.yaml file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M54pGG6vldqfY4i4cv1kzg.png" /><figcaption>Conda details for experiment which failed as serving endpoint: Image by author</figcaption></figure><p>If you’re using AutoML to get your experiments up and running, you’ll need to go to the experiment notebook to get the source code (which is automatically generated for you by Databricks/AutoML).</p><p>Navigate to your registered model &gt; select the version &gt; click the link to the source run &gt; open the source notebook.</p><p>Now that we have the code, we can update our conda environment logged to the model.</p><p>These AutoML-generated scripts are structured to contain an <em>objective </em>function which is then run across various parameter configurations. Look for the cell containing the following code to see the model generated as a result:</p><pre>best_result = trials.best_trial[&quot;result&quot;]<br>model = best_result[&quot;model&quot;]<br>mlflow_run = best_result[&quot;run&quot;]</pre><p>Insert a new cell to log a new version of the model with your conda environment details. Use <em>log_model </em>to save this to your experiments.</p><pre>cd = {<br>        &quot;channels&quot;: [&quot;conda-forge&quot;],<br>        &quot;dependencies&quot;: [<br>            &quot;python=3.9.5&quot;,<br>            &quot;pip&lt;=21.2.4&quot;,<br>            &quot;pandas==1.5.3&quot;,<br>            {<br>                &quot;pip&quot;: [<br>                    &quot;mlflow&lt;3,&gt;=2.1&quot;,<br>                    &quot;cffi==1.15.0&quot;,<br>                    &quot;cloudpickle==2.0.0&quot;,<br>                    &quot;psutil==5.8.0&quot;,<br>                    &quot;scikit-learn==1.0.2&quot;,<br>                    &quot;typing-extensions==4.1.1&quot;,<br>                    &quot;databricks-automl-runtime==0.2.15&quot;,<br>                    &quot;holidays==0.18&quot;,<br>                    &quot;category-encoders==2.5.1.post0&quot;,<br>                    &quot;xgboost==1.7.2&quot;,<br><br>                ]<br>            },<br>        ],<br>        &quot;name&quot;: &quot;mlflow-env&quot;,<br>    }<br>mlflow.sklearn.log_model(model, &quot;dbfs:/databricks/mlflow-tracking/1675317468748639&quot;, conda_env=cd)</pre><p>Once done, repeat the process of registering the model and update your serving endpoint to point to this version of the model. Once deployed, it should look as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qEhdLRJdPX5ATs55fwN_FQ.png" /><figcaption>Model deployed successfully: Image by author</figcaption></figure><p><strong><em>Important Note: Your endpoint deployment logs will only display the first missing module. In the event of several missing modules, I’ve found it best to look at the other experiment run generated as part of your notebook and ensure that ALL dependencies in your new conda_env are found in the source experiment’s conda_env yaml file.</em></strong></p><p>With that, we’re ready to start using the REST API to query the model!</p><p>Thanks for reading — if you’ve seen any issues with this post or have further comments, please let me know.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dd24692a598e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering MLOps: A 6 month learning plan with MLflow]]></title>
            <link>https://medium.com/@mc12338/mastering-mlops-a-6-month-learning-plan-with-mlflow-cec1988c4c70?source=rss-d1970f1605f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/cec1988c4c70</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[mlflow]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Matt Collins]]></dc:creator>
            <pubDate>Wed, 12 Apr 2023 19:01:48 GMT</pubDate>
            <atom:updated>2023-04-12T19:01:48.357Z</atom:updated>
            <content:encoded><![CDATA[<h4>A structured learning path for MLOps</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*c92VMzP5Jc1RD4FN" /><figcaption>Photo by <a href="https://unsplash.com/@goumbik?utm_source=medium&amp;utm_medium=referral">Lukas Blazek</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I’m currently brushing up on my machine learning and MLOps skills. Despite playing around with machine learning for many years, I’ve only recently started putting it into practice and have found that implementations which follow best practices are quite different to your experiments that involve importing a CSV from your local machine.</p><p>The resources available for learning this growing field increases day by day as new technologies and considerations are introduced (technical or ethical, for example). It is easy to get overwhelmed by trying to do everything at once.</p><p>As such, I’ve decided to put together a focused learning plan to tackle topics one at a time and keep my development on track. I’ll be using MLflow in Databricks as part of this process, which I’ll mention a bit later in this post.</p><p>I usually prefer writing technical content but wanted to share my plan to keep myself accountable in my development and, maybe, help some others along the way. This blog joins me a couple of months into this journey, so I will share what I’ve learnt along the way and what is still to come.</p><h3>Tools</h3><p>I’ve been using MLflow for a while now and it is a great product to apply machine learning and follow MLOps design patterns.</p><p>Recently, MLflow have released <a href="https://mlflow.org/docs/latest/recipes.html">MLflow recipes</a>, providing a framework of reproducible steps for deploying, monitoring and maintaining a model. I will use these steps as a guideline to my learning path, before spending more time looking at MLflow recipes as a whole, hopefully in time for this to be released as General Availability!</p><p>Since I primarily work in the Microsoft Azure cloud, I’ll be using Azure Databricks for my environment. This has the benefit of ML compute clusters with MLflow installed by default, so no need to worry about configuring Python, MLflow or associated libraries on my machine. It also provides an easy way for me to replicate different environments for MLOps tasks.<br>One area of particular interest for me is integrating machine learning with production workloads and I use Databricks a lot in my day-to-day data engineering tasks. Again, it is a natural choice for this work.</p><h3>Plan</h3><p>With that being said, let’s jump into the month-by-month plan…</p><h4>Month 1 ✅</h4><p>Objectives: <br>1. The Big picture and ML lifecycle.</p><ul><li>How can machine learning benefit businesses? What case studies show the value when implemented?</li><li>What is the difference between hype and the value of machine learning in practice?</li><li>What is the current state of MLOps and how can we incorporate machine learning into our current data solutions?</li></ul><p>2. Improve knowledge of Python APIs.</p><ul><li>Investigate some simple notebook examples, focusing on processes rather than complex algorithms.</li><li>Review documentation to see what is available out of the box.</li></ul><p>Key results:</p><ul><li>Increased knowledge of machine learning use cases.</li><li>Blog: <a href="https://medium.com/@mc12338/5-quick-tips-to-improve-your-mlflow-model-experimentation-dae346db825">5 quick tips to improve your MLflow model experimentation</a>.</li><li><a href="https://github.com/MattPCollins/MLflowExperimentation">Notebook</a> of code snippets.</li></ul><p>Outcomes:</p><ul><li>This month had a focus on understanding the value of machine learning. As a consultant looking to help clients get value from their data, it is important to understand what positive impact machine learning activities can bring.</li><li>The documentation has helped simplify using MLflow as a tool and identify some functionality that other users aren’t using (or at least aren’t writing about).</li></ul><h4>Month 2 ✅</h4><p>Topics:<br>1. The MLOps process for deploying from Development to Production.</p><ul><li>What are the recommended approaches when using models across different environments?</li><li>What does the deployment process look like?</li><li>What are some of the challenges with moving models across environments?</li></ul><p>Key results:</p><ul><li>Blog: <a href="https://medium.com/@mc12338/automate-ml-model-retraining-and-deployment-with-mlflow-in-databricks-ad29f6146f80">Automate ML model retraining and deployment with MLflow in Databricks</a>.</li><li><a href="https://github.com/MattPCollins/MLflowRetrainModel">Notebooks</a> I can use as templates for deployments.</li></ul><p>Outcomes:</p><ul><li>This has been an area of interest to me for a while, so it has been great to spend more time reading various sources and understanding <em>why</em> the best practices are as they are.</li><li>Reproducible template for deploying/redeploying models to use in other projects.</li></ul><h4>Month 3 ⏳</h4><p>Topics:<br>1. Model serving options.</p><ul><li>What are batch and real-time inference?</li><li>What are the real-world use cases of either for businesses?</li><li>What is the cost of real-time serving (MLflow and Databricks)?</li></ul><p>2. Incorporate machine learning into Data Lakehouse.</p><ul><li>What are the design patterns to follow?</li><li>Architecture do’s and don’ts.</li></ul><p>Key results:</p><ul><li>Blog idea: How to integrate MLflow into your data product/solution.</li><li>Blog idea: Store ML predictions in your Data Lake Architecture with Databricks and MLflow.</li><li>Blog idea: Power BI and MLflow real-time serving endpoint for on-demand predictions for report users.</li></ul><h4>Month 4 🔜</h4><p>Topics:<br>1. Model Explainability.</p><ul><li>Explaining explainability in simple terms.</li><li>Get to grips with SHAP and other explainability metrics.</li></ul><p>2. Intro to Feature Stores.</p><ul><li>What are Feature Stores and how can businesses benefit from them?</li><li>Are Feature Stores relevant for businesses of all sizes?</li><li>How do Feature Stores fit into a Data Solution architecture?</li><li>What are typical Feature Stores for Regression, Classification and Time Series models?</li></ul><p>Key results:</p><ul><li>Blog idea: Understand your data better with SHAP in MLflow.</li><li>Visualise simple ML architecture including Feature Store.</li></ul><h4>Month 5 🔜</h4><p>Topics<br>1. ML model monitoring.</p><ul><li>What are the key things to watch when monitoring your machine learning models?</li><li>What are the different types of drift?</li><li>What actions can be taken to ensure model performance remains good?</li><li>Are explainability metrics a good addition to model monitoring reports?</li></ul><p>2. Feature Stores continued: FeatureStoreClient API.</p><ul><li>What is required to put a Feature Store into practice?</li><li>How does this integrate with the Deploy Code approach of productionising ML solutions?</li><li>How can I make model interaction with Feature Stores robust for changing models and additional features being added?</li></ul><p>Key results:</p><ul><li>ML monitoring dashboard.</li><li>Blog idea: Statistics to monitor in your ML model monitoring dashboard.</li><li>Feature Store notebooks for Classification, Regression and Time Series models.</li></ul><h4>Month 6 🔜</h4><p>Topics:<br>1. MLflow Recipes.</p><ul><li>What are the pros and cons of MLflow Recipes.</li><li>Have I understood all components of MLflow recipes, and the ML lifecycle?</li><li>Do MLflow recipes create a robust machine learning solution ready for your data solution?</li></ul><p>Key results:</p><ul><li>Blog idea: A breakdown of the MLflow Recipes components in 5 minutes.</li></ul><h3>Closing Thoughts</h3><p>This post has allowed me to reflect on the step-by-step process I need to take to both build out my MLOps knowledge and understanding of how to implement a full machine learning solution. While the topics and results may change slightly over the coming months, I’ll use this post to keep myself accountable for studying and reflecting on what I’ve learned through tangible outputs — such as blogs or code.</p><p>I’m staying away from getting too heavily into the algorithms themselves in this journey. As a consultant, demonstrating a successful process for implementing ML solutions is more useful to clients who are starting their ML journeys. I’ll get more into the mathematics of it all further down the line.</p><p>Is there anything you’d add to this study plan? Let me know your thoughts and questions in the comments!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cec1988c4c70" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automate ML model retraining and deployment with MLflow in Databricks]]></title>
            <link>https://medium.com/data-science/automate-ml-model-retraining-and-deployment-with-mlflow-in-databricks-ad29f6146f80?source=rss-d1970f1605f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad29f6146f80</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[mlflow]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Matt Collins]]></dc:creator>
            <pubDate>Wed, 15 Mar 2023 16:20:23 GMT</pubDate>
            <atom:updated>2023-04-10T15:32:56.114Z</atom:updated>
            <content:encoded><![CDATA[<h4>Efficiently manage and deploy production models with MLflow</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dWxyy04eIm6YUKVo" /><figcaption>Photo by <a href="https://unsplash.com/es/@karsten116?utm_source=medium&amp;utm_medium=referral">Karsten Winegeart</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Getting a working machine learning model deployed for user consumption is a great achievement. We see <a href="https://www.gartner.com/en/newsroom/press-releases/2018-02-13-gartner-says-nearly-half-of-cios-are-planning-to-deploy-artificial-intelligence">statistics</a> showing that machine learning models often fail to make it into production, whether this is due to insufficient data, lack of direction or other reasons.</p><p>Models which do make it into production still face many challenges as they require consistent attention in the forms of monitoring and retraining to ensure the insights they aim to provide are up-to-date and accurate over time.</p><p>This blog aims to help streamline the model retraining process with MLflow while providing background on the recommended approach.</p><h3>Why retrain a production model?</h3><p>Model retraining is the process of giving our production model access to the latest data to run up-to-date predictions. Depending on the sophistication of our system, we might perform this retraining under various scenarios, such as:</p><ul><li>At regular intervals: Such as on a weekly schedule.</li><li>Upon certain criteria: <a href="https://www.databricks.com/blog/2019/09/18/productionizing-machine-learning-from-deployment-to-drift-detection.html">Data drift</a> hitting a threshold condition may result in us retraining to accommodate a changing data landscape.</li><li>Upon registration of a new model: Our Data Scientists have found a more accurate model which has been approved to go live.</li></ul><h3>MLflow</h3><p>Model retraining falls under the Machine Learning Operations (MLOps) process and MLflow is a great tool that helps simplify this in an iterative fashion, allowing smoother delivery with reproducible executions.</p><p>If you are new to MLflow there are many detailed resources available online, but I recommend starting with their <a href="https://mlflow.org/">website</a> to see their offerings — I’ve included the components offered below as an introduction.</p><p>We will be making use of the MLflow <a href="https://mlflow.org/docs/latest/tracking.html">Tracking</a> component to log our retraining experiment runs and the <a href="https://mlflow.org/docs/latest/model-registry.html">Model Registry</a> component to ensure deployment is seamless and mitigates the need for downtime in our production environment.</p><h3>Pre-requisites</h3><p>As we’re talking about retraining, we’ve made the assumption that you already have a model (and the data you wish to predict against) in production. If you do not, and you wish to use MLflow to achieve this, I’ve provided <a href="https://github.com/MattPCollins/MLflowRetrainModel/blob/main/MLflowDeployModel.ipynb">this notebook</a> to get started. We’ll review this process and code shortly in the “Deploy an initial model to production” section.</p><p>Out of convenience, we’ll be using a Databricks workspace with ML compute cluster since this provides us with a managed environment with all of the required packages installed, the MLflow interface and embedded and a spark environment to assist with any big data queries through parallel processing, if required.</p><p>Databricks is available through most cloud providers — I’ll be using Microsoft Azure. If running MLflow locally, then ensure all of the relevant packages are installed and an MLflow tracking server is set up.</p><p>I’ve broken down these requirements below:</p><ul><li>A Databricks workspace running a ML compute cluster, simulating a production environment.</li><li>Source data: In production, I would expect this to be tables in our Data Warehouse or Lake. We’ll just use the Diabetes dataset in the Scikit-learn package for this example.</li><li>An existing Machine Learning model which we will save as a production model in the Model Registry (code sample below).</li></ul><h3>Deploy an initial model to production</h3><p>The following code block shows an experiment run for a Ridge Regression model. For the full notebook, see this link in my <a href="https://github.com/MattPCollins/MLflowRetrainModel/blob/main/MLflowDeployModel.ipynb">GitHub Repo</a>. This example is designed to provide you with a “production” model which you can base the model retraining process against.</p><pre># Start MLflow run for this experiment<br><br># End any existing runs<br>mlflow.end_run()<br><br>with mlflow.start_run() as run:<br>  # Turn autolog on to save model artifacts, requirements, etc.<br>  mlflow.autolog(log_models=True)<br><br>  diabetes_X = diabetes.data<br>  diabetes_y = diabetes.target<br>  <br>  # Split data into test training sets, 3:1 ratio<br>  diabetes_X_train, diabetes_X_test, diabetes_y_train, diabetes_y_test = train_test_split(diabetes_X, diabetes_y, test_size=0.25, random_state=42)<br><br>  alpha = 1<br>  solver = &#39;cholesky&#39;<br>  regr = linear_model.Ridge(alpha=alpha,solver=solver)<br><br>  regr.fit(diabetes_X_train, diabetes_y_train)<br><br>  diabetes_y_pred = regr.predict(diabetes_X_test)<br>  <br>  # Log desired metrics<br>  mlflow.log_metric(&quot;mse&quot;, mean_squared_error(diabetes_y_test, diabetes_y_pred))<br>  mlflow.log_metric(&quot;rmse&quot;, sqrt(mean_squared_error(diabetes_y_test, diabetes_y_pred)))<br>  mlflow.log_metric(&quot;r2&quot;, r2_score(diabetes_y_test, diabetes_y_pred))</pre><p>Use the MLflow API commands to push this to production in your Model Registry.</p><pre>model_uri = &quot;dbfs:/databricks/mlflow-tracking/&lt;&gt;/&lt;&gt;/artifacts/model&quot;<br>desc = &#39;Initial model deployment&#39;<br>new_run_id = run.info.run_id<br>client.create_model_version(name, model_uri, new_run_id, description=desc)<br>version = client.search_model_versions(&quot;run_id=&#39;{}&#39;&quot;.format(new_run_id))[0].version<br>client.transition_model_version_stage(name, version, &quot;Production&quot;)</pre><h3>Finding the right approach: Deploy Code vs Deploy Model</h3><p>To understand what our retraining notebook might look like, we need to understand the approach taken. Microsoft discusses two deployment patterns which I’ve summarised below. More in-depth information can be found <a href="https://learn.microsoft.com/en-us/azure/databricks/machine-learning/mlops/deployment-patterns">here</a>.</p><h4>Deploy Code</h4><ul><li>ML artifacts are packaged as code from deployment to production.</li><li>Version control and testing can be implemented.</li><li>The deployment environment is reproduced in production, reducing the risk of production issues.</li><li>Production models are trained against the production data.</li><li>Additional deployment complexity.</li></ul><h4>Deploy Model</h4><ul><li>Standalone artifacts (Machine Learning model) are deployed to production.</li><li>Flexibility to deploy to different types of environments or integrate with different services.</li><li>Simplicity in the deployment process.</li><li>Fast deployment time with easy versioning.</li><li>Changes and enhancements to Feature engineering, monitoring, etc need to be managed separately.</li></ul><p>We’ll be taking the recommended approach of Deploying Code. This lends itself nicely to what we are trying to achieve: we can take our Production-ready, stakeholder-approved script and use this for the retraining process.</p><p><strong>We’re not changing any parameter values in our model: we are simply retraining the model against the latest data.</strong></p><p>Start by setting the experiment and model that we’re retraining.</p><pre># Import packages<br>from mlflow.client import MlflowClient<br><br># Set the experiment name to an experiment in the shared experiments folder<br>mlflow.set_experiment(&quot;/diabetes_regression_lab&quot;)<br><br>client = MlflowClient()<br><br># Set model name<br>name = &#39;DiabetesRegressionLab&#39;</pre><p>Load the dataset. As mentioned, we’re simplifying this requirement by using the Diabetes dataset from the scikit-learn package. In reality, this might be a select statement against a table.</p><pre># Load the diabetes dataset<br>diabetes = datasets.load_diabetes()</pre><p>In this case we’re simulating this model retraining occurring after some data change, possibly a few days later. To simulate this time delta, the current registered production model has been trained on a subset of the data (1.) and we’re using the full dataset to show further data being added over time (2.)</p><pre># 1. Mimic results from a week ago, used by our registered production model<br>diabetes_X = diabetes.data[:-20]<br>diabetes_y = diabetes.target[:-20]<br><br># 2. Dataset as of point of retraining, used in our latest experiment run<br>diabetes_X = diabetes.data<br>diabetes_y = diabetes.target</pre><p>Note that there might be a use case to remove some historical data from your training data set, should data drift be detected.</p><p>Once loaded, kick off an MLflow run and start training. This will follow the usual process of splitting the data, training, predicting and comparing to the test dataset.</p><pre># Start MLflow run for this experiment: This is similar to your experimentation script<br>mlflow.end_run()<br><br>with mlflow.start_run() as run:<br>    # Turn autolog on to save model artifacts, requirements, etc.<br>  mlflow.autolog(log_models=True)<br>  <br>  # Split data into test training sets, 3:1 ratio<br>  diabetes_X_train, diabetes_X_test, diabetes_y_train, diabetes_y_test = train_test_split(diabetes_X, diabetes_y, test_size=0.25, random_state=42)<br><br>  alpha = 1<br>  solver = &#39;cholesky&#39;<br>  regr = linear_model.Ridge(alpha=alpha,solver=solver)  <br><br>  regr.fit(diabetes_X_train, diabetes_y_train)<br><br>  diabetes_y_pred = regr.predict(diabetes_X_test)<br>  <br>  # Log desired metrics<br>  mlflow.log_metric(&quot;mse&quot;, mean_squared_error(diabetes_y_test, diabetes_y_pred))<br>  mlflow.log_metric(&quot;rmse&quot;, sqrt(mean_squared_error(diabetes_y_test, diabetes_y_pred)))<br>  mlflow.log_metric(&quot;r2&quot;, r2_score(diabetes_y_test, diabetes_y_pred))</pre><h3>Validation Criteria</h3><p>It is good practice to use validation criteria to determine if our new model performs at least as well as the existing one before replacing it in production. This process helps to ensure the new model is reliable and minimises the risk of performance degradation. In this case, I’m simply using the mse, rmse and r2 values as my validation metrics.</p><p>Now that we’ve logged a new run, we can compare this to the run which is currently in production.</p><pre># Collect latest run&#39;s metrics<br>new_run_id = run.info.run_id<br>new_run = client.get_run(new_run_id)<br>new_metrics = new_run.data.metrics<br><br># Collect production run&#39;s metrics<br>prod_run_id = client.get_latest_versions(name, stages=[&quot;Production&quot;])[0].run_id<br>prod_run = client.get_run(prod_run_id)<br>prod_metrics = prod_run.data.metrics<br><br># Collate metrics into DataFrame for comparison<br>columns = [&#39;mse&#39;,&#39;rmse&#39;,&#39;r2&#39;]<br>columns = [&#39;version&#39;] + [x for x in sorted(columns)]<br>new_vals = [&#39;new&#39;] + [new_metrics[m] for m in sorted(new_metrics) if m in columns]<br>prod_vals = [&#39;prod&#39;] + [prod_metrics[m] for m in sorted(prod_metrics) if m in columns]<br>data = [new_vals, prod_vals]<br><br>metrics_df = pd.DataFrame(data, columns=columns)<br>metrics_df</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/793/1*a6s7sZYSi7X69yVofBsMog.png" /><figcaption>Metrics DataFrame: Image by author</figcaption></figure><p>This is a simple example where our model uses a Ridge regression algorithm. In reality, our model may be composite and use a hyperparameter search space, comparing multiple algorithms and automatically deciding the “best” one to use under complex validation criteria. The same concept can be applied, and the successful model published to the Model Registry for consumption.</p><h3>Promote in the Model Registry</h3><p>We can then use the following code to automatically move this run to Production in the Model Registry, upon meeting these validation criteria specified:</p><pre># Retrieve validation variables from the metrics DataFrame<br>new_mse = metrics_df[metrics_df[&#39;version&#39;] == &#39;new&#39;][&#39;mse&#39;].values[0]<br>new_rmse = metrics_df[metrics_df[&#39;version&#39;] == &#39;new&#39;][&#39;rmse&#39;].values[0]<br>new_r2 = metrics_df[metrics_df[&#39;version&#39;] == &#39;new&#39;][&#39;r2&#39;].values[0]<br><br>prod_mse = metrics_df[metrics_df[&#39;version&#39;] == &#39;prod&#39;][&#39;mse&#39;].values[0]<br>prod_rmse = metrics_df[metrics_df[&#39;version&#39;] == &#39;prod&#39;][&#39;rmse&#39;].values[0]<br>prod_r2 = metrics_df[metrics_df[&#39;version&#39;] == &#39;prod&#39;][&#39;r2&#39;].values[0]<br><br># Check new model meets our validation criteria before promoting to production<br>if (new_mse &lt; prod_mse) and (new_rmse &lt; prod_rmse) and (new_r2 &gt; prod_r2):<br>  model_uri = &quot;dbfs:/databricks/mlflow-tracking/&lt;&gt;/&lt;&gt;/artifacts/model&quot;<br>  print(&#39;run_id is: &#39;, new_run_id)<br>  <br>  desc = &#39;This model uses Ridge Regression to predict diabetes.&#39;<br><br>  client.create_model_version(name, model_uri, new_run_id, description=desc)<br>  to_prod_version = client.search_model_versions(&quot;run_id=&#39;{}&#39;&quot;.format(new_run_id))[0].version<br>  to_archive_version = client.search_model_versions(&quot;run_id=&#39;{}&#39;&quot;.format(prod_run_id))[0].version<br>  <br>  # Transition new model to Production stage<br>  client.transition_model_version_stage(name, to_prod_version, &quot;Production&quot;)<br>  <br>  # Wait for the transition to complete<br>  new_prod_version = client.get_model_version(name, to_prod_version)<br>  while new_prod_version.current_stage != &quot;Production&quot;:<br>      new_prod_version = client.get_model_version(name, to_prod_version)<br>      print(&#39;Transitioning new model... Current model version is: &#39;, new_prod_version.current_stage)<br>      time.sleep(1)<br><br>  # Transition old model to Archived stage<br>  client.transition_model_version_stage(name, to_archive_version, &quot;Archived&quot;)<br><br>else:<br>  print(&#39;no improvement&#39;)<br>  </pre><p>In our case, all validation criteria were hit so the model has been pushed to production in the model registry. Any utilisation of the model, batch or real-time, will now be against this version.</p><p>You will hopefully start to notice the benefit of the Deploy Code approach at this point, as it gives us complete control over the scripts used to retrain a model and also accommodate changing validation criteria for automatic redeployment.</p><p><strong>The full notebook can be accessed in my </strong><a href="https://github.com/MattPCollins/MLflowRetrainModel/blob/main/MLflowRetrainModel.ipynb"><strong>GitHub Repository</strong></a><strong> if you wish to see the code in full.</strong></p><h3>Utilisation</h3><p>We can continue using inference pipelines &amp; REST API calls to access the model as before, being careful to update any changes to schema if this has taken place.</p><p>This is something we can factor into our deployment code to the model registry as a reminder to prevent this from being a breaking change and any end-user impact.</p><h3>Notes/Considerations</h3><p>As of MLflow 2.0, <a href="https://www.mlflow.org/docs/latest/recipes.html">Recipes</a> is an experimental feature (at the time of writing) which provides a streamlined approach to some of this functionality, with particular reference to the validation criteria. I expect further development of Recipes to give users a well-structured and repeatable approach for the model deployment and retraining elements of the ML lifecycle.</p><p>It is also worth noting that MLOps is still relatively new and we certainly see that in the variety of approaches different businesses and users are taking in implementing ML solutions. Standardising the approach is still a work in progress, with additional components such as explainability and monitoring being more prevalent in the lifecycle.</p><p>Thanks for reading and let me know if you have any questions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad29f6146f80" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/automate-ml-model-retraining-and-deployment-with-mlflow-in-databricks-ad29f6146f80">Automate ML model retraining and deployment with MLflow in Databricks</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 Quick Tips to Improve Your MLflow Model Experimentation]]></title>
            <link>https://medium.com/data-science/5-quick-tips-to-improve-your-mlflow-model-experimentation-dae346db825?source=rss-d1970f1605f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/dae346db825</guid>
            <category><![CDATA[tips-and-tricks]]></category>
            <category><![CDATA[mlflow]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Matt Collins]]></dc:creator>
            <pubDate>Mon, 13 Mar 2023 20:29:09 GMT</pubDate>
            <atom:updated>2023-04-03T22:04:06.751Z</atom:updated>
            <content:encoded><![CDATA[<h4>Use the MLflow python API to drive better model development</h4><p>MLflow is a fantastic way to speed up your machine learning model development process through its powerful experimentation component. This enables Data Scientists to log the best algorithms and parameter combinations and rapidly iterate model development.</p><p>This blog aims to show users how to get the most out of MLflow experiments. We will focus on the start_run() and its parameters, which can enhance your experimentation process. Additionally, we will cover the search_runs() function, which provides an expansive view of your experimentation history and enables greater flexibility in analysis.</p><p>If you are new to MLflow, I suggest taking a look at the MLflow <a href="https://mlflow.org/">site</a>, <a href="https://mlflow.org/docs/latest/quickstart.html">documentation</a>, some blog posts or tutorial videos before jumping into this blog.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*po1FIA9lFjlMPWdg" /><figcaption>Photo by <a href="https://unsplash.com/@adrienconverse?utm_source=medium&amp;utm_medium=referral">Adrien Converse</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4><strong>mlflow.start_run()</strong></h4><p>Most of these tricks are parameters of the start_run() function. We call this function to initiate our experiment run, and it becomes the active run where we can log parameters, metrics, and other information.</p><p>This is the function I use most in MLflow and the one which offers the most instantaneous value to users.</p><h4>1. run_id</h4><p>The run_id is a UUID which is specific to each experiment run. Once a run has been initiated, it is not possible to overwrite properties such as the model type or parameter values. However, you can use the run_id to log additional values retrospectively, such as metrics, tags, or a description.</p><pre># Start MLflow run for this experiment<br><br># End any existing runs<br>mlflow.end_run()<br><br>with mlflow.start_run() as run:<br>    # Turn autolog on to save model artifacts, requirements, etc.<br>    mlflow.autolog(log_models=True)<br><br>    print(run.info.run_id)<br><br>    diabetes_X = diabetes.data<br>    diabetes_y = diabetes.target<br><br>    # Split data into test training sets, 3:1 ratio<br>    (<br>        diabetes_X_train,<br>        diabetes_X_test,<br>        diabetes_y_train,<br>        diabetes_y_test,<br>    ) = train_test_split(diabetes_X, diabetes_y, test_size=0.25, random_state=42)<br><br>    alpha = 0.9<br>    solver = &quot;cholesky&quot;<br>    regr = linear_model.Ridge(alpha=alpha, solver=solver)<br><br>    regr.fit(diabetes_X_train, diabetes_y_train)<br><br>    diabetes_y_pred = regr.predict(diabetes_X_test)<br><br>    # Log desired metrics<br>    mlflow.log_metric(&quot;mse&quot;, mean_squared_error(diabetes_y_test, diabetes_y_pred))<br>    mlflow.log_metric(<br>        &quot;rmse&quot;, sqrt(mean_squared_error(diabetes_y_test, diabetes_y_pred))<br>    )</pre><p>In this case, we may also want to log our coefficient of determination (r²) value for this run:</p><pre>with mlflow.start_run(run_id=&quot;3fcf403e1566422493cd6e625693829d&quot;) as run:<br>    mlflow.log_metric(&quot;r2&quot;, r2_score(diabetes_y_test, diabetes_y_pred))</pre><p>The run_id can either be extracted by print(run.info.run_id) from the previous run, or by querying mlflow.search_runs(), but more on that later.</p><h4>2. experiment_id</h4><p>You can set the experiment you want a run to log to in a few different ways in MLflow. The first command sets the experiment for all subsequent runs to “mlflow_sdk_test”.</p><pre>mlflow.set_experiment(&quot;/mlflow_sdk_test&quot;)</pre><p>This can also be configured on a run-by-run basis through the experiment_id parameter.</p><pre>my_experiment = mlflow.set_experiment(&quot;/mlflow_sdk_test&quot;)<br>experiment_id = my_experiment.experiment_id</pre><p>This value can then be reused when passed to start_run():</p><pre># End any existing runs<br>mlflow.end_run()<br><br>with mlflow.start_run(experiment_id=experiment_id):<br>    # Turn autolog on to save model artifacts, requirements, etc.<br>    mlflow.autolog(log_models=True)<br><br>    print(run.info.run_id)<br><br>    diabetes_X = diabetes.data<br>    diabetes_y = diabetes.target<br><br>    # Split data into test training sets, 3:1 ratio<br>    (<br>        diabetes_X_train,<br>        diabetes_X_test,<br>        diabetes_y_train,<br>        diabetes_y_test,<br>    ) = train_test_split(diabetes_X, diabetes_y, test_size=0.25, random_state=42)<br><br>    alpha = 0.8<br>    solver = &quot;cholesky&quot;<br>    regr = linear_model.Ridge(alpha=alpha, solver=solver)<br><br>    regr.fit(diabetes_X_train, diabetes_y_train)<br><br>    diabetes_y_pred = regr.predict(diabetes_X_test)<br><br>    # Log desired metrics<br>    mlflow.log_metric(&quot;mse&quot;, mean_squared_error(diabetes_y_test, diabetes_y_pred))<br>    mlflow.log_metric(<br>        &quot;rmse&quot;, sqrt(mean_squared_error(diabetes_y_test, diabetes_y_pred))<br>    )<br>    mlflow.log_metric(&quot;r2&quot;, r2_score(diabetes_y_test, diabetes_y_pred))</pre><h4>3. run_name</h4><p>When you specify the name of your run, you have greater control over the naming process than relying on the default names generated by MLflow. This enables you to establish a consistent naming convention for experiment runs, similar to how you might manage other resources in your environment.</p><pre># Start MLflow run for this experiment<br><br># End any existing runs<br>mlflow.end_run()<br><br># Explicitly name runs<br>today = dt.today()<br><br>run_name = &quot;Ridge Regression &quot; + str(today)<br><br>with mlflow.start_run(run_name=run_name) as run:<br>    # Turn autolog on to save model artifacts, requirements, etc.<br>    mlflow.autolog(log_models=True)<br><br>    print(run.info.run_id)<br><br>    diabetes_X = diabetes.data<br>    diabetes_y = diabetes.target<br><br>    # Split data into test training sets, 3:1 ratio<br>    (<br>        diabetes_X_train,<br>        diabetes_X_test,<br>        diabetes_y_train,<br>        diabetes_y_test,<br>    ) = train_test_split(diabetes_X, diabetes_y, test_size=0.25, random_state=42)<br><br>    alpha = 0.5<br>    solver = &quot;cholesky&quot;<br>    regr = linear_model.Ridge(alpha=alpha, solver=solver)<br><br>    regr.fit(diabetes_X_train, diabetes_y_train)<br><br>    diabetes_y_pred = regr.predict(diabetes_X_test)<br><br>    # Log desired metrics<br>    mlflow.log_metric(&quot;mse&quot;, mean_squared_error(diabetes_y_test, diabetes_y_pred))<br>    mlflow.log_metric(<br>        &quot;rmse&quot;, sqrt(mean_squared_error(diabetes_y_test, diabetes_y_pred))<br>    )<br>    mlflow.log_metric(&quot;r2&quot;, r2_score(diabetes_y_test, diabetes_y_pred))</pre><p>However, please be aware that run_name is not a unique constraint in MLflow. This means that you could have multiple experiments (with unique run IDs) sharing the same name.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4TskhVlRknzt0lTWFdOvHg.png" /><figcaption>MLflow experiment table view — Duplicate run names: Image by author</figcaption></figure><p>This means that every time you execute a new run in a <em>with </em>statement, it will create a new experiment of the same name, rather than append details to this run.</p><p>This brings us nicely to the next parameter.</p><h4>4. nested</h4><p>You may be familiar with nested experiment runs if you’ve run the scikit-learn functionGridSearchCV to perform hyperparameter optimisation.</p><p>Nested experiments look something like the following in MLflow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pnur5zno9XbCJj8bPK1pbg.png" /><figcaption>MLflow experiment table view — nested experiments: Image by author</figcaption></figure><p>Note that the metrics here are saved against the parent run, which returns the best values recorded by the child runs. The child run values themselves are blank.</p><p>While nested experiments are excellent for evaluating and logging parameter combinations to determine the best model, they also serve as a great logical container for organizing your work. With the ability to group experiments, you can compartmentalize individual data science investigations and keep your experiments page organized and tidy.</p><pre># End any existing runs<br>mlflow.end_run()<br><br># Explicitly name runs<br>run_name = &quot;Ridge Regression Nested&quot;<br><br>with mlflow.start_run(run_name=run_name) as parent_run:<br>    print(parent_run.info.run_id)<br><br>    with mlflow.start_run(run_name=&quot;Child Run: alpha 0.1&quot;, nested=True):<br>        # Turn autolog on to save model artifacts, requirements, etc.<br>        mlflow.autolog(log_models=True)<br><br>        diabetes_X = diabetes.data<br>        diabetes_y = diabetes.target<br><br>        # Split data into test training sets, 3:1 ratio<br>        (<br>            diabetes_X_train,<br>            diabetes_X_test,<br>            diabetes_y_train,<br>            diabetes_y_test,<br>        ) = train_test_split(diabetes_X, diabetes_y, test_size=0.25, random_state=42)<br><br>        alpha = 0.1<br>        solver = &quot;cholesky&quot;<br>        regr = linear_model.Ridge(alpha=alpha, solver=solver)<br><br>        regr.fit(diabetes_X_train, diabetes_y_train)<br><br>        diabetes_y_pred = regr.predict(diabetes_X_test)<br><br>        # Log desired metrics<br>        mlflow.log_metric(&quot;mse&quot;, mean_squared_error(diabetes_y_test, diabetes_y_pred))<br>        mlflow.log_metric(<br>            &quot;rmse&quot;, sqrt(mean_squared_error(diabetes_y_test, diabetes_y_pred))<br>        )<br>        mlflow.log_metric(&quot;r2&quot;, r2_score(diabetes_y_test, diabetes_y_pred))</pre><p>Should you need to add to this nested run, then specify the parent run’s run_id in subsequent executions as a parameter, appending further child runs.</p><pre># End any existing runs<br>mlflow.end_run()<br><br>with mlflow.start_run(run_id=&quot;61d34b13649c45699e7f05290935747c&quot;) as parent_run:<br>    print(parent_run.info.run_id)<br>    with mlflow.start_run(run_name=&quot;Child Run: alpha 0.2&quot;, nested=True):<br>        # Turn autolog on to save model artifacts, requirements, etc.<br>        mlflow.autolog(log_models=True)<br><br>        diabetes_X = diabetes.data<br>        diabetes_y = diabetes.target<br><br>        # Split data into test training sets, 3:1 ratio<br>        (<br>            diabetes_X_train,<br>            diabetes_X_test,<br>            diabetes_y_train,<br>            diabetes_y_test,<br>        ) = train_test_split(diabetes_X, diabetes_y, test_size=0.25, random_state=42)<br><br>        alpha = 0.2<br>        solver = &quot;cholesky&quot;<br>        regr = linear_model.Ridge(alpha=alpha, solver=solver)<br><br>        regr.fit(diabetes_X_train, diabetes_y_train)<br><br>        diabetes_y_pred = regr.predict(diabetes_X_test)<br><br>        # Log desired metrics<br>        mlflow.log_metric(&quot;mse&quot;, mean_squared_error(diabetes_y_test, diabetes_y_pred))<br>        mlflow.log_metric(<br>            &quot;rmse&quot;, sqrt(mean_squared_error(diabetes_y_test, diabetes_y_pred))<br>        )<br>        mlflow.log_metric(&quot;r2&quot;, r2_score(diabetes_y_test, diabetes_y_pred))</pre><p>One thing to note about this approach is that your metrics will now be logged against each <strong>child </strong>run.</p><h4>5. mlflow.search_runs()</h4><p>This trick is using the search_runs()function.</p><p>This function allows us to programmatically query the experimentation GUI, and the results are returned in a tabular format that is easy to understand and manipulate.</p><p>In the below example, we can select specific fields from the runs in our experiment and load them into a Pandas DataFrame. Notice that the available columns greatly exceed those available in the experiments GUI!</p><pre># Create DataFrame of all runs in *current* experiment<br>df = mlflow.search_runs(order_by=[&quot;start_time DESC&quot;])<br><br># Print a list of the columns available<br># print(list(df.columns))<br><br># Create DataFrame with subset of columns<br>runs_df = df[<br>    [<br>        &quot;run_id&quot;,<br>        &quot;experiment_id&quot;,<br>        &quot;status&quot;,<br>        &quot;start_time&quot;,<br>        &quot;metrics.mse&quot;,<br>        &quot;tags.mlflow.source.type&quot;,<br>        &quot;tags.mlflow.user&quot;,<br>        &quot;tags.estimator_name&quot;,<br>        &quot;tags.mlflow.rootRunId&quot;,<br>    ]<br>].copy()<br>runs_df.head()</pre><p>As this is a Pandas DataFrame, we can add columns that may be useful for analysis:</p><pre># Feature engineering to create some additional columns<br>runs_df[&quot;start_date&quot;] = runs_df[&quot;start_time&quot;].dt.date<br>runs_df[&quot;is_nested_parent&quot;] = runs_df[[&quot;run_id&quot;,&quot;tags.mlflow.rootRunId&quot;]].apply(lambda x: 1 if x[&quot;run_id&quot;] == x[&quot;tags.mlflow.rootRunId&quot;] else 0, axis=1)<br>runs_df[&quot;is_nested_child&quot;] = runs_df[[&quot;run_id&quot;,&quot;tags.mlflow.rootRunId&quot;]].apply(lambda x: 1 if x[&quot;tags.mlflow.rootRunId&quot;] is not None and x[&quot;run_id&quot;] != x[&quot;tags.mlflow.rootRunId&quot;]else 0, axis=1)<br>runs_df</pre><p>If we want to aggregate the result set to provide information of runs over time, we can use:</p><pre>pd.DataFrame(runs_df.groupby(&quot;start_date&quot;)[&quot;run_id&quot;].count()).reset_index()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/165/1*cFnKc-MxW-0nZ1OHL0EOmA.png" /><figcaption>Output of the above query: Image by author</figcaption></figure><p>The automatic tags.estimator_name field allows us to review how many runs have been tested for each algorithm.</p><pre>pd.DataFrame(runs_df.groupby(&quot;tags.estimator_name&quot;)[&quot;run_id&quot;].count()).reset_index()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/219/1*fg4HuXioN4OlVmVzYYvKJQ.png" /><figcaption>Output of the above query: Image by author</figcaption></figure><p>Given this is a DataFrame, we can export the data for any reporting requirements to give the required visibility to users who may not have access to the workspace, and compare across workspaces.</p><h3>Closing thoughts</h3><p>These are just a few examples of how to extend your use of MLflow’s functions and parameters in your experimentation process, but there are many more available in the python API.</p><p>Hopefully, this post has inspired you to explore some of the available functions and parameters and see if they can benefit your model development process. For additional information, refer to the API <a href="https://mlflow.org/docs/latest/python_api/index.html">documentation</a> and experiment with different configurations to find the best fit for your needs.</p><p>If you’re currently using any functions or parameters that I’ve not mentioned in this post, please let me know in the comments!</p><p>All code can be found in my <a href="https://github.com/MattPCollins/MLflowExperimentation">GitHub Repo</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dae346db825" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/5-quick-tips-to-improve-your-mlflow-model-experimentation-dae346db825">5 Quick Tips to Improve Your MLflow Model Experimentation</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>