<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Anastasia Prokaieva on Medium]]></title>
        <description><![CDATA[Stories by Anastasia Prokaieva on Medium]]></description>
        <link>https://medium.com/@anastasia.prokaieva?source=rss-21aed4db6339------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*gE75gFtS1YPS15Otz3byMQ.jpeg</url>
            <title>Stories by Anastasia Prokaieva on Medium</title>
            <link>https://medium.com/@anastasia.prokaieva?source=rss-21aed4db6339------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 04:44:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@anastasia.prokaieva/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Universal Agent Illusion: Why Treating Your SaaS as an “MCP Data Pipe” is a Costly Mistake]]></title>
            <link>https://medium.com/@anastasia.prokaieva/the-universal-agent-illusion-why-treating-your-saas-as-an-mcp-data-pipe-is-a-costly-mistake-94e4f209c7c4?source=rss-21aed4db6339------2</link>
            <guid isPermaLink="false">https://medium.com/p/94e4f209c7c4</guid>
            <category><![CDATA[salesforce]]></category>
            <category><![CDATA[saas]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Anastasia Prokaieva]]></dc:creator>
            <pubDate>Wed, 15 Apr 2026 09:04:32 GMT</pubDate>
            <atom:updated>2026-04-15T09:04:47.261Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RpkOoQpFP1xYn2u9PtX7Xg.jpeg" /><figcaption>Author used Nano Banana to make it live!</figcaption></figure><p>Walk into any enterprise architecture board meeting today, and you’ll hear a very compelling narrative pushed by the major hyperscalers: <em>“SaaS is dead. In the era of Generative AI, your application is nothing more than a bucket of data wrapped in an API.”</em></p><p>Their proposed architecture? Build a “Universal Supervisor Agent” on their generic cloud infrastructure. Connect it to your enterprise applications using standards like the Model Context Protocol (MCP) or simple API tool calling. Feed their LLMs your data, and let the model figure out the logic.</p><p>To a cloud-native developer, this sounds like liberation. Why be locked into Salesforce, SAP, or Workday workflows when you can just use them as “dumb data pipes” for a central, all-knowing AI?</p><p>Because an enterprise application is not just data and an API. It is the culmination of decades of hardened domain expertise, deterministic guardrails, and operational truth. Extracting that intelligence into an external LLM doesn’t liberate your enterprise; it <strong>creates a multimillion-dollar re-platforming project </strong>disguised as AI innovation.</p><p>Let’s look at how the “SaaS is dead” crowd defends the Universal Agent model — and why those architectural arguments fall apart when deployed in high-velocity, real-time business operations.</p><h3>Argument 1: The Latency &amp; Data Freshness Myth</h3><p><strong>The Hyperscaler Pitch:</strong> <em>“We don’t query 15-year-old SaaS APIs in real-time. We stream all your CRM, ERP, and WMS data into a high-speed Lakehouse, vector-index it, and the AI queries it in milliseconds. Latency is solved.”</em></p><blockquote><strong>The “Yes, But…” Reality:</strong> Yes, you can dump everything into a data lake to speed up queries. But a data lake is an <em>analytical</em> store, not an <em>operational</em> one.</blockquote><p>Imagine a customer asking an AI agent: <em>“Where is my order? If it’s not coming today, I want a refund.” </em>If your data lake syncs every 15 minutes, or even every 5 minutes, it is operationally blind. If that order shipped two minutes ago, your lightning-fast Lakehouse agent will confidently tell the customer it’s still processing.</p><p>Platform-native AI (like Salesforce Agentforce) operates on the operational truth <em>right now</em>. Through zero-copy architectures (like Data Cloud), the AI reads the live transactional state without moving the data. It doesn’t rely on the last batch sync; it acts on reality. You get sub-2-second response times without sacrificing transactional accuracy.</p><h3>Argument 2: The Security &amp; Identity Illusion</h3><p><strong>The Hyperscaler Pitch:</strong> <em>“We don’t need to rebuild your SaaS security model. We use federated Identity and OAuth. When the user asks a question, the MCP tool passes their specific token. If they don’t have access to a record in the CRM, the API rejects it. The LLM never sees forbidden data.”</em></p><blockquote><strong>The “Yes, But…” Reality:</strong> Yes, OAuth will prevent a data breach. But API denial is not a business workflow — it’s a dead end.</blockquote><p>Native SaaS security isn’t just about <em>blocking</em> data; it’s about <em>shaping the workflow</em>. If an external agent requests a customer’s contract via MCP and gets an HTTP 403 Forbidden error, the LLM simply fails, apologizes to the user, or worse, hallucinates an answer.</p><p>A platform-native agent inherently understands the context of <em>why</em> data is restricted. It knows the user’s role hierarchy. Instead of crashing, it seamlessly pivots to the native escalation workflow: <em>“I see your contract has special Platinum terms. Let me instantly transfer this chat to your dedicated account executive who has clearance to process this.”</em> You cannot replicate the elegance of native role hierarchies and sharing rules via external API rejection codes.</p><h3>Argument 3: “Legacy Logic” vs. Deterministic Guardrails</h3><p><strong>The Hyperscaler Pitch:</strong> <em>“SaaS platforms are bloated with 20 years of rigid, click-heavy workflows that humans hate. We don’t want to replicate your business logic; we want the LLM to dynamically reason and generate better, cross-system logic on the fly.”</em></p><blockquote><strong>The “Yes, But…” Reality:</strong> Dynamically generating business logic on the fly is an un-auditable compliance nightmare.</blockquote><p>In the consulting or manufacturing worlds, you operate under strict SLAs, compliance mandates, and liability risks. If an external LLM “reasons” that it should issue a $5,000 refund because the customer sounds angry, who is liable?</p><p>What developers call “legacy technical debt,” the business calls “deterministic guardrails.” Those rigid SaaS workflows exist to prevent fraud, ensure compliance, and protect margins. When you rip the AI out of the platform, you lose those guardrails. Agentforce keeps the AI grounded inside the platform’s business rules, ensuring that Generative AI acts with strict operational integrity.</p><h3>Argument 4: The TCO Trap (Build vs. Buy)</h3><p><strong>The Hyperscaler Pitch:</strong> <em>“Packaged SaaS AI agents are expensive. If we build it on AWS or Google, we own the IP, and LLM token costs are just pennies.”</em></p><blockquote><strong>The “Yes, But…” Reality:</strong> Tokens are cheap. AI Engineering talent and maintenance are not.</blockquote><p>Choosing the Universal Agent route is a construction project. Hyperscalers give you a box of extremely powerful Lego bricks (APIs, CLI tools, MCP protocols) and promise you can build a Ferrari. But MIT research shows that internal AI builds succeed only 33% of the time.</p><p>You will spend 9 to 12 months paying high-tier Python developers to build orchestrators, manage rate limits, update API endpoints every time a vendor changes a schema, and chase down hallucination bugs. With Agentforce, you get the Ferrari on day one. You trade the illusion of “cheap tokens” for predictable, outcome-based pricing and a time-to-value measured in weeks, not years.</p><h3>The Path Forward: Cohabitation, Not Replacement</h3><p>None of this means hyperscaler AI platforms don’t have a crucial place in the enterprise. They absolutely do.</p><p>If you are a manufacturing firm running R&amp;D simulations on material science, or a financial firm running custom algorithmic trading models, you need the raw compute and bespoke flexibility of Google, AWS, or Azure. That is your <em>differentiation</em>.</p><p>But for your core operational flow — customer service, talent acquisition, sales enablement — <strong>the brain belongs in the application.</strong></p><p>The future of enterprise architecture isn’t a single monolithic Universal Agent that treats your business like a dumb pipe. It is a <strong>Domain-Based Agentic Architecture</strong>. Use Agentforce to autonomously manage your CRM domain with speed, native security, and out-of-the-box reliability. When a complex process touches unstructured data or a bespoke internal system, your native agents can securely collaborate with hyperscaler agents via secure Agent-to-Agent (A2A) protocols.</p><blockquote>As Amazon’s CTO Werner Vogels says, <em>“There is no compression algorithm for experience.”</em></blockquote><blockquote>Don’t spend millions trying to compress 25 years of operational maturity into a prompt. Use the engine that already exists; just let AI drive the car.</blockquote><p>P.S&gt; Inspired by the Architect team of Salesforce France and all my customers and friends who are challenging us everyday!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=94e4f209c7c4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Chaos to Order: I Built a GenAI Butler to Organize My Google Drive]]></title>
            <link>https://medium.com/@anastasia.prokaieva/from-chaos-to-order-i-built-a-genai-butler-to-organize-my-google-drive-515529e05dfe?source=rss-21aed4db6339------2</link>
            <guid isPermaLink="false">https://medium.com/p/515529e05dfe</guid>
            <category><![CDATA[gemini]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Anastasia Prokaieva]]></dc:creator>
            <pubDate>Thu, 23 Oct 2025 14:06:05 GMT</pubDate>
            <atom:updated>2025-10-23T14:06:05.325Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BKrhwHefdJSNSaCjyKyePQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6dd9vT45CuHIOsDSJQhqsg.png" /><figcaption>Images generated with Imagen4 by the author.</figcaption></figure><p>Let’s be honest. If you’re anything like me, your Google Drive is a digital attic. It’s a beautiful, chaotic mess of untitled documents, vacation photos from 2017, half-finished project proposals, and at least seven files named final_report_v2_final_FINAL.pdf.</p><p>As a GenAI engineer, I spend my days building intelligent systems. Yet, my own digital backyard was a jungle. The irony was not lost on me. I was tired of manually sifting through files, so I decided to do what any engineer would do: over-engineer a solution.</p><p>What started as a simple cleanup script evolved into something much cooler, a series of AI agents that can <strong><em>analyze</em></strong>, <strong><em>recommend</em></strong>, and even <strong><em>act</em></strong> on my behalf to bring order to the chaos.</p><p>Let me take you on that journey.</p><h3>A Quick Word on Butlers…</h3><p>You might be thinking, “A ‘butler’? Isn’t that a bit dramatic for a Python script?”</p><p>Fair point. But here’s the difference:</p><ul><li>A <strong>script</strong> is a power tool. It does exactly what you tell it to, and if you point it at the wrong wall, you’re getting a hole in the wrong wall.</li><li>An <strong>AI assistant</strong> is like an intern. You can give it simple, well-defined tasks, but you still have to manage it closely.</li><li>A <strong>butler</strong> understands <strong><em>intent</em></strong>. You don’t tell Jeeves <strong><em>how</em></strong> to organize the library; you just say, “Jeeves, the library is a disaster,” and he intuits the need for dusting, sorting, and alphabetizing.</li></ul><p>My agent, armed with its tools and Gemini’s brain, is a digital Jeeves. It translates my vague complaints (“this file is in the wrong place!”) into precise actions. And unlike an intern, it never drinks all the coffee.</p><h3>Phase 1: The Foundation — A Humble Duplicate Finder</h3><p>Every great project starts with solving the most annoying problem first. For me, that was duplicates. I knew I had gigabytes of identical files scattered across different folders.</p><blockquote>I built path_cleaner. This isn’t the “AI” part, but it’s the bedrock — the essential plumbing that any more advanced system would need.</blockquote><p>Its core job is:</p><ol><li><strong>Authentication:</strong> Securely connect to the Google Drive API. The classic token.json / credentials.json dance using Google’s OAuth libraries. Nothing fancy, but it has to be solid.</li><li><strong>Path Navigation:</strong> The Google Drive API thinks in terms of file IDs, not human-readable paths like /Work/Reports/2023. I wrote a helper function, get_folder_by_path, that translates a familiar path into the ID the API needs. This was crucial for making the tool user-friendly.</li></ol><pre># Snippet from path_cleaner.py<br>def get_folder_by_path(service, path):<br>    # ... logic to split path and traverse from &#39;root&#39; down ...<br>    for part in path_parts:<br>        query = f&quot;name=&#39;{part}&#39; and &#39;{current_parent_id}&#39; in parents&quot;<br>        # ... API call to find the folder&#39;s ID ...<br>        current_parent_id = target_folder[&quot;id&quot;]<br>    return target_folder</pre><p><strong>3. Duplicate Detection:</strong> The magic here is the md5Checksum. Google Drive automatically calculates this unique hash for most files. If two files have the same MD5 hash, they are bit-for-bit identical. My script scans a folder, groups files by their hash, and flags any group with more than one file.</p><p>Armed with this script, I could run a command like python path_cleaner.py — path “/Downloads/Old” — action move and watch it automatically find and move duplicate files to a dedicated folder.</p><p>Satisfying? Absolutely. But it only solved one problem. The <em>structure</em> of my Drive was still a mess.</p><h3>Phase 2: The Analyst — Giving the Script Eyes and a Brain</h3><p>The duplicate finder was great, but it was blind. It couldn’t tell me that I had a dozen “Untitled” documents or that my project files were mixed in with memes. I didn’t just need a janitor; I needed a consultant. This is where Generative AI, specifically Google’s Gemini, comes into play.</p><blockquote>I created agent_drive, a “read-only” agent whose job is to see, understand, and advise.</blockquote><p>Here’s how it works:</p><p><strong>1. The Senses (Data Gathering):</strong> The agent first needs to perceive its environment. I wrote a function, generate_drive_map, to recursively scan my Drive (or a specific folder) and build a simple text-based map of the folder structure.</p><pre>- Work/<br>  -  - Project_Alpha/<br>  -  -  - design_mockup_v1.png<br>  -  -  - meeting_notes.gdoc<br>  -  - marketing_plan_Q3.pdf<br>- Photos/<br>  -  - vacation_2022.jpg<br>  -  - IMG_5082.HEIC<br>- random_screenshot.png<br>- Meeting Notes.gdoc</pre><p><strong>2. The Brain (Gemini’s Analysis):</strong> This simple text map is gold for a Large Language Model. I fed this map, along with any duplicate file info, directly into Gemini with a carefully crafted prompt.</p><p>Prompt Engineering is key here. You can’t just throw data at the model. You have to give it a role and clear instructions.</p><p>My Prompt to Gemini (Simplified):</p><pre><br>You are an expert file organization assistant. Your goal is to help users tidy up their digital space. Analyze the following file and folder structure from a user&#39;s Google Drive.<br><br># FOLDER STRUCTURE:<br>{drive_map_str}<br><br>Based on this structure, please provide actionable recommendations. <br>Focus on:<br>* Suggested Folder Structure: Propose new folders to better categorize the files.<br>* Naming Conventions: Suggest if file names could be improved for consistency.<br>* Redundancy or Junk: Point out potentially redundant files or folders.<br>* Overall Strategy: Give a high-level summary.<br><br>Provide your response in clear, easy-to-read Markdown format.</pre><p><strong>3. The Output (The Report):</strong> The agent then takes Gemini’s response and generates a beautiful Markdown report, saving it as a local file. Suddenly, I had an AI consultant telling me things like:</p><ul><li><em>“I notice several files like random_screenshot.png and Meeting Notes.gdoc in your root directory. Consider creating a _Inbox or _Unsorted folder for these to process later.”</em></li><li><em>“You have multiple photos with inconsistent names. I recommend a YYYY-MM-DD-Event.jpg format for better sorting.”</em></li><li><em>“The folders Project_Alpha and Client_Work seem related. Perhaps they could be nested under a parent Projects folder.”</em></li></ul><p>This was a game-changer. The agent couldn’t <em>do</em> anything yet, but its insights were exactly what I needed.</p><h3>Phase 3: The Butler — An Agent That Takes Action</h3><blockquote>Analysis is great, but I’m lazy. I don’t want to just be told <em>what</em> to do; I want an agent who can <em>do it for me</em>.</blockquote><p>This led to the final evolution: agent_act, an interactive agent that uses Gemini’s function calling capabilities. This is where things get really cool. The AI can now operate “tools” that I define.</p><p><strong>1. The Toolkit:</strong> I created a DriveTools class in Python. This class contains simple, well-defined functions that perform single actions, like move_file and rename_file. These methods are the bridge between the AI’s intent and the Google Drive API.</p><pre># Snippet from agent_adv.py<br>class DriveTools:<br><br>    def __init__(self, service):<br>        self.service = service<br><br>    def move_file(self, file_path: str, destination_path: str):<br>        &quot;&quot;&quot;<br>        Moves a file or folder from one location to another in Google Drive.<br>        &quot;&quot;&quot;<br>        print(f&quot;Attempting to move &#39;{file_path}&#39; to &#39;{destination_path}&#39;...&quot;)<br>        # ... logic to find file and destination IDs and call the Drive API ...<br>        return f&quot;Successfully moved file.&quot;<br><br>    def rename_file(self, file_path: str, new_name: str):<br>        &quot;&quot;&quot;<br>        Renames a file or folder in Google Drive.<br>        &quot;&quot;&quot;<br>        # ... logic to find file ID and call the Drive API ...<br>        return f&quot;Successfully renamed file.&quot;</pre><p><strong>2. The Intelligent Core:</strong> I initialized the Gemini model and told it about my toolkit. The google.generativeai library makes this incredibly easy.</p><pre># Giving the model its tools<br>model = genai.GenerativeModel(<br>    model_name=&quot;gemini-2.5-flash&quot;,<br>    tools=[drive_tools.move_file, drive_tools.rename_file]<br>)<br>chat = model.start_chat(enable_automatic_function_calling=True)</pre><p>The enable_automatic_function_calling=True part is pure magic. It means the model, when it decides a tool is needed, will have the library automatically execute my Python function with the correct arguments and feed the result back into the conversation.</p><p><strong>3. The Conversation:</strong> Now, I can have a natural language conversation with my Drive.</p><pre>You: Move the file &#39;/Photos/vacation_2022.jpg&#39; to a folder called &#39;/Archive/Holidays&#39;<br>Agent: (Thinks: The user wants to move a file. The move_file tool is perfect for this. I need file_path=&#39;/Photos/vacation_2022.jpg&#39; and destination_path=&#39;/Archive/Holidays&#39;.)<br>(The library automatically calls my drive_tools.move_file(...) function)<br>Agent: Successfully moved &#39;vacation_2022.jpg&#39; to &#39;/Archive/Holidays&#39;.<br>You: Please rename &#39;/Docs/report_v1.pdf&#39; to &#39;Project_Phoenix_Final_Report.pdf&#39;<br>Agent: (Thinks: Renaming task. I&#39;ll use the rename_file tool.)<br>(Library calls drive_tools.rename_file(...))<br>Agent: Done! I&#39;ve renamed the file to &#39;Project_Phoenix_Final_Report.pdf&#39;.</pre><h3>The Road Ahead</h3><p>This journey from a simple script to an interactive AI butler has been incredibly fun and, more importantly, useful. My Drive is already looking cleaner.</p><p>But I’m not done yet. The next steps are to expand the agent’s toolkit:</p><ul><li><strong>create_folder:</strong> Let the agent build the structures it recommends.</li><li><strong>delete_file:</strong> A powerful but dangerous tool that will need careful confirmation.</li><li><strong>summarize_document:</strong> Combine the Drive API with the Gemini API to read and summarize .gdoc files on the fly.</li><li><strong>recursive_actions:</strong> “Go through my entire /Downloads folder and organize everything by file type.”</li></ul><p>Building this has reinforced a key belief: the future of productivity lies in creating personalized AI agents that can understand our context and act on our behalf. We’re moving beyond simple scripts and into a world of conversational automation.</p><p>Now, if you’ll excuse me, I have an agent to chat with about cleaning up my Drive storage.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=515529e05dfe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Serve Many Forecasting Models with Databricks: Part 2 — Online Store.]]></title>
            <link>https://medium.com/@anastasia.prokaieva/serve-many-forecasting-models-with-databricks-part-2-online-store-9b1c6b89bb2d?source=rss-21aed4db6339------2</link>
            <guid isPermaLink="false">https://medium.com/p/9b1c6b89bb2d</guid>
            <category><![CDATA[mlflow]]></category>
            <category><![CDATA[forecasting]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[serving]]></category>
            <category><![CDATA[feature-store]]></category>
            <dc:creator><![CDATA[Anastasia Prokaieva]]></dc:creator>
            <pubDate>Wed, 10 Jul 2024 08:31:48 GMT</pubDate>
            <atom:updated>2024-07-10T09:26:53.283Z</atom:updated>
            <content:encoded><![CDATA[<h3>Serve Many Forecasting Models with Databricks: Part 2 — Online Store.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ay_hmerIeMWh-F0Yj2KaZg.jpeg" /><figcaption>Generated using Adobe Firefly GenAI model by the author “Databricks supergirl running in real-time”.</figcaption></figure><p>T his blog is supporting material to the talk delivered during the Data &amp; AI Summit 2024, <a href="https://www.databricks.com/dataaisummit/session/many-model-forecasting-real-time-databricks-online-store">“Many Model Forecasting in Real-Time”</a>, run annually by Databricks in San Francisco.</p><p>The first part of the series can be found <a href="https://medium.com/@anastasia.prokaieva/serve-many-forecasting-models-with-databricks-model-serving-at-once-80d5e3f32943">here</a>, where I have explained how to train thousands of individual time-series models using native distributing training on Spark, leveraging technology like Delta and MLflow to further serve them in real-time using Databricks Serving.</p><p>In this second part, we are going to explore how you can lift certain limitations and facilitate data augmentation using the Databricks Feature Engineering Client and Online Store to later combine your freshly arriving features with historical features to serve your forecast in real time. We will also explore how to use your models as features instead of container packaging method.</p><p>Please be aware that the techniques used across the blog can be applied to various use cases, not only forecasting. The main scenario for such an application would be real-time serving with additional low-latency features joined with frequent updates.</p><h3><strong>Let’s talk about common pitfalls</strong></h3><p>While working on the predictive analytics use cases, we generally face a few common issues, and this usually becomes even harder when we are trying to move from the batch process into real-time. Let’s cover a few that I have spotted while working on the real-time forecasting use case myself:</p><ul><li>How do I bring my historical features to my model during the inference in real-time?</li><li>How do I update my models in real-time after the retraining has happened?</li><li>How to backtest a real-time serving model?</li><li>How to join real-time data with historical series?</li></ul><p>In the <a href="https://medium.com/@anastasia.prokaieva/serve-many-forecasting-models-with-databricks-model-serving-at-once-80d5e3f32943">previous blog</a>, we learned how to fit and predict hundreds of thousands of individual forecasting models using Spark, MLFlow, and Delta and served these models as a packaged artifact through the container via the Databricks Serving. By doing so you can hit a few limitations:</p><ul><li>memory limit (as of today June 2024 Databricks does not offer different container sizes for CPU, and it’s limited to 4 GB)</li><li>required to recreate a container when new model versions arrived</li><li>MLOps is not as obvious as we cannot guarantee what was published under a container with 100% certainty</li></ul><p>To solve these, this second part of the blog will focus on moving forward with the previous solution and providing an updated scenario to overcome limitations and make it more robust.</p><h4>Let’s talk about the use case</h4><p>This and the previous blog is focusing on the following scenario:</p><blockquote><em>Retailer that operates hundreds of thousands stores and want to bring operational forecasting of sales across all stores with all the available data in real-time taking into account metadata available.</em></blockquote><p>Together with the team within Databricks(thank you for the collaboration Tarik Boukherissa), we have created various architectures, and here is one of the examples you can get inspired from the following one that was used by many of our customers:</p><figure><img alt="Architecture example" src="https://cdn-images-1.medium.com/max/1024/1*pRsK-SPU6BA-nyEVY1lBag.png" /><figcaption>Architecture example followed in our scenario.</figcaption></figure><p>In this part, we are not focusing on ingesting data or creating API backend or training models(if you want to know how we did it please check Part1 of this blog series, but we will be focusing on preparing our data and writing them to the Databricks Feature Store powered by Delta, publishing out features into low-latency database(Databricks Online Store), creating new MLFlow Pyfunc Wrapper in order to call our features during the inference in real-time and serving our forecasts.</p><h3>Part 1 — Let’s start from Features</h3><p>In our use case, our main target is Sales, and we would like to create a forecast with a 30-day horizon per Store_ID. We would usually be required to use external regressors that may impact our forecast, such as weather, promotions, average sales per week, competitions, etc.</p><p>The first question here is how to track our feature list (say you want to experiment with various features and conduct A/B testing). This is where a tool such as a feature store comes in handy. There are various feature stores, but the main concept was well described in <a href="https://medium.com/data-for-ai/feature-store-101-b964373891c4">this blog</a>, so I highly recommend you check it out if you have never heard of a feature store.</p><p>Databricks Feature Store was recently refreshed and is now named Feature Engineering with Unity Catalog(UC). If you have already used the Feature Store Client of Databricks without UC, it’s very simple to migrate: just follow <a href="https://docs.databricks.com/en/machine-learning/feature-store/uc/feature-tables-uc.html#language-Python">this link</a>. If you are not using it at all or not using UC, the following recommendations can be implemented using other tools, but all the code examples will be executable and possible only with the UC. Once your data is under a managed Delta Table, the only thing you need to do to use FS Client is to set a constraint on a primary key.</p><pre>ALTER TABLE ${catalog}.{schema}.YOUR_TABLE_NAME <br>  ALTER COLUMN column_name SET NOT NULL;<br>ALTER TABLE ${catalog}.{schema}.YOUR_TABLE_NAME <br>  ADD CONTRAINT YOUR_TABLE_NAME_pk PRIMARY KEY(column_name);</pre><p>If you do not have any Delta Table created prior to your work, you can simply prepare your features and write your data directly with the FS Client API into a table.</p><pre>from databricks.feature_engineering import FeatureEngineeringClient<br><br>fe = FeatureEngineeringClient()<br><br>feature_table = fe.create_table(<br>  name=&#39;catalog.schema.your_fs_table&#39;,<br>  primary_keys=&#39;id&#39;,<br>  schema=features_df.schema,<br>  description=&#39;Your description&#39;<br>)<br><br>fe.write_table(<br>  name=&#39;catalog.schema.your_fs_table&#39;,<br>  df = features_df,<br>  mode = &#39;merge&#39;<br>)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*akHfqZ1yUodgJhz4FYiXhQ.png" /><figcaption>Creating multiple feature tables from raw bronze source Delta Tables.</figcaption></figure><p>Once your features are written, you are ready to start using your features and connecting their metadata to your trained models. In order to create a particular metadata per a feature set (think you may play with various features, and you would like to keep track of each of them), you have to create so-called FeatureLookup and TrainingSet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zZ95V4A5swpTSpGhDEkKaw.png" /><figcaption>Example of creating a Feature Lookup from 2 distinct FS tables where we are taking only 3 features.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*FcqgigRp6mrSu7awWCfOuw.png" /><figcaption>Once you have defined your lookup of features we need to create metadata associated with it — by creating a training set.</figcaption></figure><p>If you’d like to materialize your training_set and use it for training just call this:</p><pre># will create a Spark DF to convert into pandas do .toPandas()<br>training_df = training_set.load_df() </pre><p>Once you have trained your model, use MLFlow to associate it’s object with the used Features.</p><pre>fe.log_model(<br>      artifact_path = &quot;model&quot;,<br>      model = YOUR_MODEL_OBJECT,<br>      flavor= mlflow.pyfunc, # change your type here<br>      pip_requirements= reqs,<br>      training_set=training_set, # your training_set <br>      registered_model_name=model_name, <br>  )</pre><h4>Publish your Features</h4><p>Right now, our features exist in a table, but in order to join them in real-time with our model input, we need to publish them into a low-latency database(otherwise our delay will be too significant, but it’s of course still possible, here we would like to demonstrate a real case scenario for low-latency).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/656/1*3oUMlVTdfNlPY8TLxDNkow.png" /><figcaption>Diagram to demonstrate tables we are initially publishing.</figcaption></figure><p>Databricks (as of the time this blog was written) offers 2 methods to access your data in low latency:</p><ol><li>Serving combined with Published Online Features. This will call features from pre-created metadata(training_set) that are published to an Online Store (low-latency database) and accessible only via Serving.</li><li>Feature Serving — this is a new tool that allows you to call features published to the Online Store externally via Serving as a service. Your features are now accessible via REST API anywhere.</li></ol><p>I am going to demonstrate both of the solutions below. I will skip the part on publishing as it’s well-detailed under the documentation and also given under the <a href="https://github.com/AnastasiaProkaieva/mmf_dbx_scaling/tree/main/dais-2024-mmf-realtime">code examples</a>. I highly recommend you review in depth the Online Table documentation page regarding limitations and debugging.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DkJNXj3y__rcZVAK02WUDA.png" /><figcaption>Example of a published Online Table.</figcaption></figure><p>In order to use Feature Searving, you need to create a Spec, that later can be served via Databricks Serving:</p><pre>from databricks.sdk import WorkspaceClient<br>from databricks.sdk.service import catalog as c<br><br>w = WorkspaceClient(token=os.environ[&#39;DATABRICKS_TOKEN&#39;], <br>                    host=os.environ[&#39;DATABRICKS_HOST&#39;])<br><br>spec = c.OnlineTableSpec(<br>          source_table_full_name=table_name, <br>          primary_key_columns=pks, <br>          run_triggered={&#39;triggered&#39;: &#39;true&#39;}, <br>          timeseries_key=timeseries_key<br>          )<br>w.online_tables.create(name=online_table_name, spec=spec)</pre><p>Once you perform the operations above, you will see a Spec created under your catalog:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nscE-Uwkb9D5rTAZYIzEIQ.png" /><figcaption>Example of a feature spec.</figcaption></figure><p>Once the spec is created, you can serve your features and call them from outside Databricks using REST API or Python SDK.</p><pre>from databricks.sdk.service.serving import EndpointCoreConfigInput, ServedEntityInput<br># To serve your Features use sdk or REST or UI<br>w.serving_endpoints.create_and_wait(<br>        name=model_endpoint_name,<br>        config=EndpointCoreConfigInput(<br>          served_entities=[<br>            ServedEntityInput(<br>              entity_name = name,<br>              scale_to_zero_enabled = True,<br>              entity_version = version, <br>              workload_size=&quot;Small&quot;,<br>              environment_vars ={<br>                  &quot;DATABRICKS_TOKEN&quot;: &quot;{{secrets/your_scope/sp_token}}&quot;,<br>                  &quot;DATABRICKS_HOST&quot;: &quot;{{secrets/your_scope/sp_host}}&quot;, })]))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PjuokUiGOacQ8geuvRwVmg.png" /><figcaption>Served Features via the Online Spec.</figcaption></figure><p>A few words about updates: The Online Store is connected to your original Delta Tables; once they have been updated, you need only to run the .sync() command to update your Online Tables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cKPK1mRydcntC8KRbuwC7A.png" /><figcaption>Diagram how our features can be joined to our published under container models.</figcaption></figure><h3>Part 2 — Let’s talk models</h3><p>We have covered how to create features and how you can have the same data offline and online but also access them outside of Databricks in low latency. Now, let’s talk about how to connect features to models and how to serve them together.</p><p>In Part 1, we trained hundreds of thousands of models, serialized them(pickled), and stored them under a Delta table while keeping our parameters under tracking from MLFLow; we then later packaged them under 1 artifact and registered a final model version.</p><h4>Let’s discuss a bit about the MLOps</h4><p>In our scenario, we frequently train many models with the same parameters and code; the only things that change are the fitting object, the quality of the forecasts, and the data. In this case, using MLFlow as many of us are used to, by tracking and saving every model, becomes quite redundant. Why? Think of it that any of your forecastings should lie within a certain threshold of errors (upper and lower boundaries), and you should be able to always fit your models into the same boundaries using the same data and models with associated parameters. In this case, keeping the individual model objects or especially registering every one of them is simply useless and unnecessary. You should see all your forecasts as 1 model, and hence it’s reasonable to see them as 1 single multi-model object. However, by keeping your models under a Delta table, you can always keep their versions by simply adding a few additional columns, like version, fit_date, code commit version, etc.</p><p>As our models are stored under a Delta table in a string format, the question that comes to mind is — why not use my models the same way as features instead of publishing them all under the container?</p><p>So how will this change the code? In reality, it&#39;s not much; you just need to do the following procedure again: altering the table to add a primary key constraint, creating a feature lookup, and creating a new training set. Let’s have a look:</p><pre>fe = FeatureEngineeringClient()<br><br>feature_lookups = [ <br>  FeatureLookup(<br>      table_name=f&quot;{database_name}.main_features_sales_fs&quot;, <br>      lookup_key=[&quot;Store&quot;, &quot;Date&quot;],<br>      feature_names=[&quot;SchoolHoliday&quot;, &quot;Promo&quot;]<br>  ),<br>  FeatureLookup(<br>      table_name=f&quot;{database_name}.extra_sales_weather_fs&quot;,  <br>      lookup_key=[&quot;Store&quot;, &quot;Date&quot;],<br>      feature_names=[&quot;Mean_TemperatureC&quot;]<br>  ), <br>]<br><br>models_lookups = [FeatureLookup(<br>      table_name=f&quot;{database_name}.sales_model_table_main&quot;,  <br>      lookup_key=[&quot;Store&quot;],<br>      feature_names=[&quot;encoded_model&quot;]<br>  ),<br>]<br><br>features_set = fe.create_training_set(<br>    df=label_df,<br>    feature_lookups=feature_lookups+models_lookups,<br>    label=&#39;Sales&#39;,<br>)<br><br>features_df = features_set.load_df()</pre><p>Now we just need to put all things together.</p><h3>Part 3 — Putting things together</h3><p>Once the training set is created we just need to connect our features to our models, for this we are gonna create a few custom PyFuncs to demonstrate 3 possibilities:</p><ol><li>Serving with Online Tables where our data and models are published.</li><li>Serving with Feature Serving, where our models are published(and can be called externally to Databricks), and Online Store, where our data is published.</li><li>Serving with models under container artifact and Online store where our data is published.</li></ol><h4>1. Serving data and models with Online Tables</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YEgwfRTna_rtSBzV7Q_reA.png" /><figcaption>Diagram how our data and models are connected.</figcaption></figure><pre>class MultiModelPyfunc(mlflow.pyfunc.PythonModel):<br>    def __init__(self):<br>      super().__init__()<br><br>    def load_context(self, context):<br>      pass<br>    <br>    def process_input(self, raw_input):<br>      pass <br><br>    def predict(self, context, model_input):<br>      <br>      output_df = pd.DataFrame()<br>      for store_id, pd_store_df in model_input.groupby(&#39;Store&#39;):<br>        # getting the model from pd_store_df<br>        model_pickled = pd_store_df.iloc[0,:][&quot;encoded_model&quot;][0]<br><br>        model = pickle.loads(urlsafe_b64decode(model_pickled.encode(&quot;utf-8&quot;)))<br>        predict_df = model.predict(context=None, <br>                                     data_input = pd_store_df[[<br>                                       &quot;Store&quot;, &quot;Date&quot;, <br>                                       &quot;SchoolHoliday&quot;, &quot;Promo&quot;, <br>                                       &quot;Mean_TemperatureC&quot;<br>                                       ]])<br>        output_df = pd.concat([output_df, predict_df], ignore_index=True)<br><br>      return output_df</pre><p>To log the FS metadata and our model object we need to use MLFlow:</p><pre>fe.log_model(<br>      artifact_path = &quot;model&quot;,<br>      model = MultiModelPyfunc(), # passing my custom pyfunc<br>      flavor= mlflow.pyfunc,<br>      pip_requirements= reqs, # requirements to be installed to the container<br>      training_set=features_set, # here it&#39;s a feature set (features + models)<br>      registered_model_name=model_name, <br>      code_path = [&#39;wrapper_model.py&#39;] # as I used a class outside of the notebook, this will help to bring your custom code to the container<br>  )</pre><h4>2. Serving models through Feature Serving</h4><p>Please be aware this example is bigger because we have also tried to implement OrderedDict to store models that were already called to reduce calling time. We have not measured the performance but it’s kept to demonstrate how complex you might go with your custom PyFunc and also how you can cache your results if required.</p><p>This example is very similar to the previous one; the difference is that here, we demonstrate how to call models from a source via REST API as if it were external.</p><p>In this case, we need to create a function get_model to call our model via REST API.</p><pre>class MultiModelPyfunc(mlflow.pyfunc.PythonModel):<br><br>    def __init__(self, model_list=[], db_host = None, db_token = None):<br>      super().__init__()<br>      if db_host and db_token:<br>        self.url = f&quot;{db_host}/serving-endpoints/dais-forecast-serving-spec/invocations&quot;<br>        self.headers = {&quot;Authorization&quot;: f&quot;Bearer {db_token}&quot;, &quot;Content-Type&quot;: &quot;application/json&quot;}<br>      else:<br>        self.url = f&quot;{os.getenv(&#39;DATABRICKS_HOST&#39;)}/serving-endpoints/dais-forecast-serving-spec/invocations&quot;<br>        self.headers = {&quot;Authorization&quot;: f&quot;Bearer {os.getenv(&#39;DATABRICKS_TOKEN&#39;)}&quot;, &quot;Content-Type&quot;: &quot;application/json&quot;}<br>      <br>      # instantiate a class of the model list<br>      try:<br>        self.model_list = LimitedOrderedDict(<br>          dict(zip(model_list[&#39;Store&#39;], model_list[&#39;encoded_model&#39;]))<br>      )<br>      except Exception as e:<br>        print(f&quot;No initial model list provided: {e} ---- instantiating empty dict&quot;)<br>        self.model_list = LimitedOrderedDict()<br><br><br>    def load_context(self, context):<br>        pass<br><br>    ## An example if you need to add a data processing on the fly to your data <br>    def process_input(self, raw_input):<br>        pass <br>    <br>    def get_model(self, store_id):<br>        # formatting the data for endpoint serving call<br>        data = {&quot;dataframe_records&quot;: [{&quot;Store&quot;: store_id}]}<br>        # preparing the call for the model<br>        data_json = json.dumps(data, allow_nan=True) <br>        response = requests.request(method=&#39;POST&#39;, <br>                                    headers=self.headers, <br>                                    url=self.url, <br>                                    data=data_json)<br>        if response.status_code != 200:<br>          raise Exception(f&#39;Request failed with status {response.status_code}, {response.text}&#39;)<br>        # Preparing the result assuming we get only the first line<br>        model_artifact = response.json()[&quot;outputs&quot;][0].get(&#39;encoded_model&#39;)<br>        model_artifact_unpickled = pickle.loads(urlsafe_b64decode(model_artifact[0].encode(&quot;utf-8&quot;)))<br>        output_dict = {store_id: model_artifact_unpickled}<br>        return output_dict<br><br>    def predict(self, context, model_input):<br>      <br>        output_df = pd.DataFrame()<br>        for store_id, pd_store_df in model_input.groupby(&#39;Store&#39;):<br>        <br>          if self.model_list.get(store_id):<br>            model = self.model_list.get(store_id)<br>            predict_df = model.predict(context=None, <br>                                       data_input = pd_store_df[[<br>                                         &quot;Store&quot;, &quot;Date&quot;, <br>                                         &quot;SchoolHoliday&quot;, &quot;Promo&quot;, <br>                                         &quot;Mean_TemperatureC&quot;<br>                                         ]])<br>            output_df = pd.concat([output_df, predict_df], ignore_index=True) <br>          elif self.model_list.get(store_id) is None:<br>            # pull the model from the feature spec add it to the ordered dictionary<br>            fetched_model = self.get_model(store_id) <br>            # add the model to the dictionnary<br>            self.model_list[next(iter(fetched_model))] = next(iter(fetched_model.values()))<br>            model = self.model_list.get(store_id) <br>            predict_df = model.predict(context=None, <br>                                       data_input = pd_store_df[[<br>                                         &quot;Store&quot;, &quot;Date&quot;, <br>                                         &quot;SchoolHoliday&quot;, &quot;Promo&quot;, <br>                                         &quot;Mean_TemperatureC&quot;<br>                                         ]])<br>            output_df = pd.concat([output_df, predict_df], ignore_index=True) <br>          else:<br>            print(&quot;Model was not found, add your implementation logic for fit on the fly here if necessary&quot;)<br><br>        return output_df</pre><p>To log the model you would use the same code in this case:</p><pre>fe.log_model(<br>      artifact_path = &quot;model&quot;,<br>      model = MultiModelPyfunc(), # passing my custom pyfunc<br>      flavor= mlflow.pyfunc,<br>      pip_requirements= reqs, # requirements to be installed to the container<br>      training_set=features_set, # here it&#39;s a feature set (features + models)<br>      registered_model_name=model_name, <br>      code_path = [&#39;wrapper_model.py&#39;] # as I used a class outside of the notebook, this will help to bring your custom code to the container<br>  )</pre><h4>3. Serving models through artifact</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cKPK1mRydcntC8KRbuwC7A.png" /><figcaption>Diagram how our data and models are connected.</figcaption></figure><p>We are loading all our models under a single artifact to be brought to the container.</p><pre>class MultiModelPyfunc(mlflow.pyfunc.PythonModel):<br><br>    def __init__(self, model_list = []):<br>      # generate the model list - we could provide that if calling the class from outside mlflow<br>      self.model_list = model_list<br><br>    def load_context(self, context):<br>        # Get JSON dictionary from the artifacts for all models and load it as pd dataframe <br>        model_list = pd.DataFrame.from_records(mlflow.artifacts.load_dict(context.artifacts[&#39;model_list&#39;]))<br>        print(model_list)<br>        model_list[&quot;model_artifact&quot;] = [pickle.loads(urlsafe_b64decode(artifact.encode(&quot;utf-8&quot;))) for artifact in model_list[&#39;encoded_model&#39;]]<br>        self.model_list = model_list.set_index(&#39;Store&#39;)<br><br>    ## An example if you need to add a data processing on the fly to your data <br>    def process_input(self, raw_input):<br>        pass <br><br>    def predict(self, context, model_input):       <br>        output_df = pd.DataFrame()<br><br>        for store_id, pd_store_df in model_input.groupby(&#39;Store&#39;):<br>          print(pd_store_df.head())<br>          if store_id in self.model_list.index.to_list():<br>            model = self.model_list.loc[store_id, &#39;model_artifact&#39;]<br>            predict_df = model.predict(context=None, <br>                                       data_input = pd_store_df[[<br>                                         &quot;Store&quot;, &quot;Date&quot;, <br>                                         &quot;SchoolHoliday&quot;, &quot;Promo&quot;, <br>                                         &quot;Mean_TemperatureC&quot;<br>                                         ]])<br>            output_df = pd.concat([output_df, predict_df]) <br>          else:<br>            print(&quot;Model was not found, add your implementation logic for fit on the fly here&quot;)<br><br>        return output_df</pre><pre>fe.log_model(<br>      artifact_path = &quot;model&quot;,<br>      model = MultiModelPyfunc(),<br>      flavor= mlflow.pyfunc,<br>      pip_requirements= reqs,<br>      artifacts= artifacts, # passing all models to the artifact in memory<br>      training_set=training_set,<br>      registered_model_name=model_name,<br>      code_path = [&#39;wrapper_model.py&#39;]<br>  )</pre><h4>Final Forecast</h4><p>All 3 methods will follow the exact same scoring process and exact same serving methodology. I am going to demonstrate it on 1 of them when your models are passed as the features.</p><p>Let’s start from batch, I will be doing a prediction forecast with and without passing features.</p><ol><li>with features that are given to the serialized model object (custom PyFunc)</li></ol><pre>store_id = &quot;1&quot;<br>logged_model = f&#39;runs:/{run.info.run_id}/model/data/feature_store/raw_model&#39;<br># Getting all data apart Sales(label)<br>main_score_df_features = features_set.load_df().drop(&quot;Sales&quot;)<br><br>data_mocking_test = main_score_df_features.filter(f&quot;Store == {store_id}&quot;).toPandas()<br><br># Getting our tracked model with custom PyFunc <br>loaded_model = mlflow.pyfunc.load_model(logged_model)<br># Predicting our test set that contains only Store ID and Date <br>predict_df = loaded_model.predict(data_mocking_test)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/286/1*cnpjyxriq3W7aMEW2VqSUQ.png" /><figcaption>Results from the prediction with features given(returns only what the PyFunc should return).</figcaption></figure><p>2. without features with the registered model where features will be looked up by feature store client (features that were associated with the model with its metadata)</p><pre>model_version = &quot;3&quot;<br>store_id = &quot;1&quot;<br># Getting only Store ID and Date, no other features <br>main_score_df_no_features = features_set.load_df().select(&quot;Store&quot;,&quot;Date&quot;)<br>test_sales_no_features = main_score_df_no_features.filter(f&quot;Store == {store_id}&quot;)<br><br># FE will look for corresponding data <br>predict_df = fe.score_batch(<br>  model_uri=f&#39;models:/{model_name}/{model_version}&#39;,<br>  # it&#39;s a Spark UDF so we need to confort the output format 30 days <br>  df=test_sales_no_features.filter(<br>    (f.col(&quot;Date&quot;) &gt;= &quot;2015-01-01&quot;) &amp; <br>    (f.col(&quot;Date&quot;) &lt; &quot;2015-01-31&quot;))<br>)<br></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wWZRlHYr50Nzxtb42h3s7Q.png" /><figcaption>Results from prediction using FS client (returns features and prediction by joining them)</figcaption></figure><p>Once we have confirmed that our model is predicting correct results offline we can move into the online version. We are going to use the same code to serve our features, you only need to pass your registered_model_name and the version you would like to serve, as always you have 3 options (REST API, SDK, UI):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lAs2VJ6TF1u3kh9rT9mKjQ.png" /><figcaption>Example of the Serving UI with the results.</figcaption></figure><p>Finally, we can verify all the assets that I have under my catalog:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/478/1*TE9A13ZYHCu9MV_OS0j1Ig.png" /></figure><h3>Part 4 — How does it work?</h3><blockquote>The first time I tried using Databricks Feature Store client and Serving, it was definitely not clear how this even could work.</blockquote><p>Let’s unpack this in a simple manner. Feature Store client creates a spec, special metadata that is connected to your table where data is stored. Later we connect this spec metadata to the model metadata via the FS and MLFlow clients.</p><p>During the scoring process in batch or real-time, a DAG(direct acyclic graph) will verify and look for features to be joined with the primary key(s) and on a timestamp if any (be aware of the time stamp here; it’s the time evolution of your features; it can be arrival time, but it’s not per see time_series feature).</p><p>For real-time, if you are passing the feature in the call, the features published simply will not be joined, but the passed one will be taken instead; the time stamp is there to help you get the latest features.</p><h3>Conclusions</h3><p>Feature Store Client from Databricks significantly simplifies your MLOps process due to its native integration with managed services on the platform, like MLFlow and Serving.</p><p>With the use of the Feature Store, we can compare various feature combinations without changing our main code.</p><p>New Online Store capabilities bring a completely new toolset that allows users to simply move from batch / streaming into real-time predictive analytics without significant code revamping. The Online Store(aka low-latency database), can be used across various projects, and the learnings above can be adapted to various scenarios!</p><p>The code can be found <a href="https://github.com/AnastasiaProkaieva/mmf_dbx_scaling/tree/main/dais-2024-mmf-realtime">under this link</a> if you’d like to give it a try!</p><h4>Next Steps</h4><p>As the next steps, you could explore a few things:</p><ul><li>add Function Serving, if necessary, to calculate additional features on the fly during the call (e.g., distance, the difference in a day, average, etc.)</li><li>add monitoring capabilities to your input data, your batch, or real-time forecasts to the Databricks Monitoring tool</li><li>try to play with fit-on-the-fly options if necessary (sometimes we do not have an ID of the user or store pretrained, but we still want to perform an operation)</li><li>create a GenAI agent that would be asked to provide a forecast per store and return results to some applications.</li></ul><h4>Limitations observed</h4><p>I’d like to mention a few limitations (there are more depending on your use case, so please check the documentation of each product under limitations):</p><ul><li>container size of 4 GB of RAM; as of today, you cannot change your default CPU container size on Databricks; if you need more memory, the only way is to go to the GPU clusters that are significantly more expensive.</li><li>size limit of the char type per row under Online Table (columns of String type are limited to 64KB length), possible to weight the limit by making it a structured type (array) but you still have the maximum size of the row is 2MB</li><li>when you want to update your models under the container, you would have to deploy a copy of your container and gradually switch, a manual process that requires preparation and time, hence moving into Models as Features fits better with a time_stamp key</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9b1c6b89bb2d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Serve Many Forecasting Models with Databricks Model Serving at Once.]]></title>
            <link>https://medium.com/@anastasia.prokaieva/serve-many-forecasting-models-with-databricks-model-serving-at-once-80d5e3f32943?source=rss-21aed4db6339------2</link>
            <guid isPermaLink="false">https://medium.com/p/80d5e3f32943</guid>
            <category><![CDATA[timeseries]]></category>
            <category><![CDATA[delta-lake]]></category>
            <category><![CDATA[mlflow]]></category>
            <category><![CDATA[pandasudf]]></category>
            <category><![CDATA[model-serving]]></category>
            <dc:creator><![CDATA[Anastasia Prokaieva]]></dc:creator>
            <pubDate>Tue, 16 May 2023 18:11:48 GMT</pubDate>
            <atom:updated>2023-05-22T19:51:50.759Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CdhWAC8f2nnhiObYW1x56g.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@robynnexy?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Robynne Hu</a> on <a href="https://unsplash.com/photos/HOrhCnQsxnQ?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Time Series Forecasting is a vital area of machine learning that has become increasingly important in today’s data-driven world. It involves the use of statistical models to analyze and predict patterns in time series data, which can then be used to make informed decisions and develop effective strategies. Time Series Forecasting has a wide range of applications, including finance, healthcare, energy, retail, weather forecasting, and many others.</p><p>However, despite its importance and potential benefits, putting a Time Series Forecasting model into production in real-time can still be a complex and challenging task. This is due to a range of factors, such as the need for accurate and reliable data; the need for robust and scalable infrastructure, the need for effective model evaluation and tuning; and the need for ongoing monitoring and maintenance. Additionally, different domains and industries may have their own unique challenges and requirements, which can further complicate the process of deploying a Time Series Forecasting model in production.</p><p>Serving a machine learning model in real-time typically involves deploying the model as an API (Application Programming Interface) that can be accessed by other applications or services. There are several ways to serve a machine learning model in real-time, depending on the specific requirements and constraints of the system:</p><ol><li>Containerization</li><li>Serverless Functions</li><li>Microservices Architecture</li><li>In-Memory Data Grids</li><li>API Gateway</li></ol><p>Ultimately, the choice of approach will depend on factors such as the size and complexity of the model, the expected workload, and the available resources and infrastructure. It’s important to consider factors like security, reliability, and scalability when designing and deploying a real-time machine learning system.</p><p>Recently Databricks announced General Availability for their managed Model Serving Endpoints. This is a managed service that creates for you concurrent endpoints. Even though it’s fully managed, you still have a lot of flexibility around, such as adding your custom code, libraries, files, etc to be served by the underlying service.</p><blockquote>Databricks Model Serving brings a highly available, low-latency and serverless service for deploying models behind an API. You no longer have to deal with the hassle and burden of managing a scalable infrastructure. It’s a fully managed service that takes care of all the heavy lifting for you, eliminating the need to manage instances, maintain version compatibility, and patch versions. Endpoints automatically scale up or down to meet demand changes, saving infrastructure costs while optimizing latency performance.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wQxIN2uLVg5k_adimCttlg.png" /><figcaption><a href="https://www.databricks.com/blog/2023/03/07/announcing-general-availability-databricks-model-serving.html">https://www.databricks.com/blog/2023/03/07/announcing-general-availability-databricks-model-serving.html</a></figcaption></figure><p>It’s straightforward to serve a classic model on Databricks Model Serving. In a few words in order to serve a classic SkLearn model you would require the following:</p><ul><li>fit your SkLearn model and log it under MLFlow on the Databricks Lakehouse platform</li><li>log all necessary dependencies into conda.yml under MLFlow artifact</li><li>register your model under MLFlow Registry</li><li>enable Databricks Model Serving Endpoint with the UI or API</li></ul><p>You are set to go :)</p><h3>Simple but …</h3><p>I have been asked to serve a more complex solution such as forecasting in real-time for many models at the same time. In the beginning, I thought this would take me no longer than a classic use case. The more I was working on it the more I was realizing that this was not as trivial as I’ve been thinking initially. And here we are!</p><p>So, let’s try to do it together — serve many forecasting models under 1 endpoint in real-time on Databricks Serving Endpoint.</p><p>Here is the context that you will be able to cover while reading this blog:</p><ol><li>How to <strong>train many independent forecasting models</strong> on top of Delta table using <a href="https://www.databricks.com/blog/2020/05/20/new-pandas-udfs-and-python-type-hints-in-the-upcoming-release-of-apache-spark-3-0.html">PandasUDF of Apache Spark</a>.</li><li>How to<strong> track each model</strong> using MLFlow and Delta tables.</li><li>How to <strong>serialize all</strong> your models <strong>under a single MLFlow object</strong>.</li><li>How to <strong>prepare your input data payload </strong>to be consumed by a single Endpoint.</li></ol><p>Before you begin to read, please keep in mind I am assuming you have knowledge of MLFlow and you are familiar with concepts of tracking, logging, model registry etc. If it’s the first time you hear about MLFlow you may want to check the <a href="https://mlflow.org/docs/latest/what-is-mlflow.html">main Documentation page</a> or check this simple <a href="https://docs.databricks.com/mlflow/end-to-end-example.html">End-2-End demo example</a> created by the Databricks team or “<a href="https://youtu.be/nx3yFzx_nHI">Introducing MLflow for End-to-End Machine Learning on Databricks</a>” recording from the 2020 Spark and AI Summit.</p><h3>Part 1 — Is this really that hard?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M9zDLy-QzdYQwMyhWkSHNg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/es/@yoshginsu?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Yosh Ginsu</a> on <a href="https://unsplash.com/photos/e4usjv5lmhE?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>For this blog post, I am going to use a well-known dataset available on the Kaggle Page for forecasting competition — <a href="https://www.kaggle.com/c/rossmann-store-sales">Rossmann Dataset</a>. It’s a perfect dataset because it contains more than 1000 individual drug stores across Germany, where you require to create forecasting of sales. In this case, we would like to create so-called local model forecasting — per store.</p><blockquote>Store sales are influenced by many factors, including promotions, competition, school and state holidays, seasonality, and locality. With thousands of individual managers predicting sales based on their unique circumstances, the accuracy of results can be quite varied.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rtVAI7yHWROWdT0sFqM69g.png" /><figcaption>Example of the data subset</figcaption></figure><p>First of all, let’s talk about why would classic Pythonic methods lead to a long execution and you would better avoid it in this case. If you require forecast more than 1 independent model, probably you would do the following:</p><pre>def mocking_fct_model(df:pd.DataFrame)-&gt;pd.DataFrame: <br>    &quot;&quot;&quot;<br>    Please pay attention this is a mocking function, <br>    most of the code is not provided for simplicity reasons. <br>    &quot;&quot;&quot;<br>    fct_model.fit()<br>    predictions = fct_model.predict()<br>    return predictions <br><br>stores = df[&quot;stores&quot;].values<br>predictions_dict = {f&quot;{store_id}&quot;: mocking_fct_model(df[&quot;stores&quot;]==str(store)) for store_id in stores}</pre><p>A code example from above may seem completely fine for most data scientists or analysts. Meanwhile, this is not the most optimal way when you require to scale your solution for a thousand models.</p><h4>Let’s talk about time</h4><p>Let’s pretend for a minute that your business requires more granular forecasting, your business team has come to you and asked to create forecasting not just per store but also per state(you have 10 states). This increased the number of models you have to train from 1000 to 10000. Next week your business team came back and told you that they require you to create a forecasting model for more than 1 horizon, they require you to have forecasted for the next 10 weeks on a daily basis — that’s just increased the number of your models up to <strong>100000</strong>!!!</p><p>It’s quite simple to estimate a benchmark time how long does this take to train 100K models following a classic Pythonic approach, saying that your fit/predict takes around 5-10 sec to execute?</p><pre># ***** <br># BENCHMARK <br># *****<br>import time <br>def benchmark_calc(store_id):<br>    # This is assuming you have a realy simple model. <br>    # More complex your model, longer it takes to train.<br>    # On average models like Prophet will take 10 sec. <br>    time.sleep(5) <br>    return 1<br>    <br>benchamark_dict = {f&quot;store_id&quot;: benchmark_calc(store_id) for store_id in range(1,YOUR_AMOUNT,1)}</pre><p>Something that may look so small and perform so quickly suddenly became a real problem for you. If you execute the above code for more than 1000 models you will realize that it takes a significant amount of time &gt;~1h! Wow, more than an hour for just 1000 models, what about 100000 then? My <strong>business unit cannot wait that long</strong>. This is where distributed systems such as<strong> Apache Spark</strong> come to the rescue.</p><h3>Part 2 — Train your models with PandasUDF</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PrxT8-7--k7TK4XhZrxFbg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@ltmonster?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">shiyang xu</a> on <a href="https://unsplash.com/images/animals/panda?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>In this part, I will show you how simple it’s to distribute your Python code using PandasUDF. I have already downloaded data from Kaggle and ingested it under Delta tables(an example of how to do so is attached under the GettingDataFromKaggle Notebook on the GitHub Page).</p><p>Before we start, I would like to show you a “compute” configuration I used to perform benchmarks and run my code on the Databricks Lakehouse Platform. Please take into account that I am not trying to find the cheapest or optimal clusters in this case, I simply selected something that seems correct for me -&gt; 1 Driver and 4 Workers with 8 cores each — this will allow me to run around 32 Tasks in parallel (by default it’s 1 CPU/task unless a configuration is changed by you).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/1*QcVm7d3_2Siij5qVgNtFSw.png" /><figcaption>The Cluster Configuration I used to run everything.</figcaption></figure><p>In order to read data from Delta Tables I am using Apache Spark:</p><pre>salesDF = spark.read.table(&quot;hive_metastore.ap.hack_ap_rossmann_blog&quot;)</pre><p>I am keeping my data vertically styled not horizontally, to increase the complexity of calculations I am adding a new variable — horizon, which will grow the dataset 7 times more(I have more than 5M rows and I would require to train Stores*Horizons models &gt; 7K models):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ISO4a6P32mXzr07i0rOWkA.png" /></figure><p>Apache Spark is a powerful open-source tool that continues evolving. But there may be times when we need to be smarter. Here I am going to give you a <strong>TIP</strong> on how to accelerate your training when your dataset is skewed. You simply have to do two things, turn off AQE(<a href="https://spark.apache.org/docs/latest/sql-performance-tuning.html">Adaptive Query Executor</a>) if enabled and repartition your dataset (ideally per individual subset):</p><pre># AQE is a great tool but it may cause issues when you have skewed tasks. <br>spark.conf.set(&quot;spark.sql.adaptive.enabled&quot;, &quot;false&quot;)<br><br># It&#39;s better to repartition your dataset before to assure 1 task per store,<br># makes shuffle in advance for you.<br>stores_amount = len(train_horizons.select(&quot;Store&quot;).distinct().collect())<br>df_full = train_horizons.repartition(stores_amount, &quot;Store&quot;)<br># To run repartitioning you need to call an action.<br>df_full.count()</pre><p>The reason for repartitioning is quite simple — Spark works with a concept of data partitioning, when your dataset is skewed Spark would probably shuffle everything first and only then create some optimal plan for you to be executed. In our case, we are going to apply some Python Code — that Spark cannot optimize, in that case for us, Spark will be an orchestrator of our unique models in parallel fashion over multiple CPUs. Repartitioning will help you to prepare your data and to avoid Spark performing additional shuffling when Python Code is mapped — this and disabling AQE can significantly accelerate your process.</p><p>In order to train so many models in a distributed fashion, I am going to use <strong>groupBy(“).applyInPandas(func_to_apply, schema=output_schema)</strong> methods of PandasUDF.</p><pre>import pyspark.sql.types as t<br>import pyspark.sql.functions as F <br># PandasUDF requires a schema of the output to be provided. <br>trainReturnSchema = t.StructType([<br>  t.StructField(&quot;Store&quot;, t.StringType()),  # unique store ID<br>  t.StructField(&quot;Horizon&quot;, t.StringType()),  # unique horizon ID<br>  t.StructField(&quot;n_used&quot;, t.IntegerType()),    # number of records used in training<br>  t.StructField(&quot;model_path&quot;, t.StringType()), # path to the model for a given combination<br>  t.StructField(&quot;encode_model&quot;, t.StringType()), # encoded string of the model <br>])<br><br># Passing an experimentID to log models appropriately. <br>experiment_id = experiment_fct.experiment_id<br>with mlflow.start_run() as run:<br>  # Passing also runID to create a nested run. <br>  run_id = run.info.run_id<br><br>  # Our dataset will be subsampled with the GroupBy operation <br>  # but the dataset does not contain information about the run and experiment.<br>  # We could make new columns with a function of Spark withColumn and<br>  # we apply our pre-created function to the Spark DataFrame.<br>  modelDirectoriesDF = (df_full<br>                        .withColumn(&quot;run_id&quot;, F.lit(run_id))<br>                        .withColumn(&quot;experiment_id&quot;, F.lit(experiment_id))  <br>                        .groupby(&quot;Store&quot;, &quot;Horizon&quot;)<br>                        .applyInPandas(fit_final_model_udf, <br>                                       schema=trainReturnSchema)<br>                        .cache()<br>                       )<br># Combining our results with the original dataset by joining both.  <br>combinedDF = (df_full.join(modelDirectoriesDF, <br>              on=[&quot;Store&quot;,&quot;Horizon&quot;],<br>              how=&quot;left&quot;)<br>              )</pre><blockquote>The code above took me ~6 min (to train and write) for individual model per Store and ~23 min per Store &amp; Horizon (7805 models)— WOW!</blockquote><blockquote>Reminder: it takes more than 1 hour just to train 1000 models if we would use a classic Pythonic way with a loop.</blockquote><p>This is what my MLFlow Experiment page looks like(as you see I have unique models per Store in that case) and a corresponding object artifact that was created during the model tracking:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hZrjILUSAYRzhDN6oGszkw.png" /><figcaption>MLFlow Experiment UI</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B8O3YdwYFN3Y4OsZaofwqQ.png" /><figcaption>MLFLow Artifact Run UI</figcaption></figure><p>Now let’s actually dig a bit more into the <strong>fit_final_model_udf()</strong> creation and see the output of this function.</p><p>In order to perform individual training using PandasUDF technique, you should keep in mind a few simple things:</p><ul><li>your input data is a subset of the main dataset, selected from GroupBy operation, Apache Arrow library takes care of serializing and deserializing your code</li><li>if your model requires input data to be ordered do it inside your training function</li><li>your output types and names should correspond to the schema that was given to the applyInPandas</li><li>your output will correspond to a data frame of a subset that will be appended to the main one by applyInPandas</li></ul><p>Here is the main function that I’ve applied to a 5M rows dataset in order to train individual models per Store and Horizon:</p><pre>from datetime import date<br>import pickle<br>import json<br>from base64 import urlsafe_b64decode, urlsafe_b64encode<br><br><br>def fit_final_model_udf(df_pandas: pd.DataFrame) -&gt; pd.DataFrame:<br>    &quot;&quot;&quot;<br>    Your input DataFrame will have following columns as an input:<br><br>    &quot;Store&quot;, &quot;State&quot; -&gt; Columns on which we are going to preform a groupBy to seperate our individual datasets<br>    &quot;Date&quot;, &quot;Sales&quot;, &quot;StateHoliday&quot;, &quot;SchoolHoliday&quot; -&gt; Columns that will be used for Training<br>    &quot;run_id&quot;, &quot;experiment_id&quot; -&gt; Columns that are necessary for logging under MLFlow Artifact <br>    <br>    &quot;&quot;&quot;<br>    from ServingForecasting import wrapper_model<br>    model = wrapper_model.ForecastingModelProphet()    <br><br>    df_pandas = df_pandas.fillna(0).sort_values(&quot;Date&quot;).copy()<br>    X = df_pandas[[&quot;Store&quot;, &quot;Horizon&quot;, &quot;Date&quot;, &quot;Sales&quot;, &quot;StateHoliday&quot;, &quot;SchoolHoliday&quot;]]<br>    y = df_pandas.loc[:, [&quot;Sales&quot;]]<br><br>    store = df_pandas[&quot;Store&quot;].iloc[0]<br>    horizon = df_pandas[&quot;Horizon&quot;].iloc[0]<br>    n_used = df_pandas.shape[0]<br>    run_id = df_pandas[&quot;run_id&quot;].iloc[0]  # pulls runID to do a nested run<br>    experiment_id = df_pandas[&quot;experiment_id&quot;].iloc[0]<br>    artifact_name = f&quot;model_custom_{store}_{horizon}&quot;<br><br>    # Resume the top-level training.<br>    with mlflow.start_run(<br>        run_id=run_id, <br>        experiment_id=experiment_id<br>        ) as outer_run:<br><br>        # Create a nested run for the specific device.<br>        with mlflow.start_run(<br>            run_name=f&quot;store_{store}_{horizon}&quot;, <br>            nested=True, <br>            experiment_id=experiment_id<br>            ) as run:<br><br>            # Defining Model Pipeline here.<br>            model.fit(X)<br>            <br>            mlflow.pyfunc.log_model(<br>                artifact_path=artifact_name,<br>                python_model=model,<br>            )<br>            # Logging date of training <br>            mlflow.log_param(f&quot;model_trained&quot;, date.today())<br>            # Pay attention that the artifact you are writting <br>            # is the same as your model.<br>            artifact_uri = f&quot;runs:/{run.info.run_id}/{artifact_name}&quot;<br>            <br>           <br>    # We are going to encode our model. <br>    # This is a TRICK how to reduce future inference time, <br>    # and will help us in our next step to wrap all in one.<br>    model_encoder = str(urlsafe_b64encode(pickle.dumps(model)).decode(&quot;utf-8&quot;))<br>    <br>    # Create a return pandas DataFrame that matches the schema above.<br>    returnDF = pd.DataFrame(<br>        [[store, horizon, n_used, artifact_uri, model_encoder]],<br>        columns=[&quot;Store&quot;, &quot;Horizon&quot;, &quot;n_used&quot;, &quot;model_path&quot;, &quot;encode_model&quot;],<br>        )<br>    return returnDF</pre><h3>Part 3 — Create a custom Wrapper for your models</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cBBDCuJf_gXCzMGOD8jVIw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@izzyfisch_?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Isabella Fischer</a> on <a href="https://unsplash.com/photos/GuO8rAZb3-c?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Now that we have trained all 7805 individual models, we have to figure out how we can use them at once when needed.</p><p>In order to serve all your models at once, we have to create a unique model object that will be serialized under MLFlow Artifact. But how to do that?</p><blockquote><em>Let’s first have a look at the final MLFlow logging, you will spot a parameter under the logging method</em>: <strong>artifacts = “/dbfs/tmp/ap/artifact.json”</strong></blockquote><pre>from mlflow.tracking import MlflowClient<br>from mlflow.models.signature import infer_signature<br><br>client = MlflowClient()<br><br>model_serving_name = &quot;multimodel-serving-fct-custom-wrapper&quot;<br><br>with mlflow.start_run() as run:<br>    model_info = mlflow.pyfunc.log_model(<br>        &quot;augmented-fct-model-custom&quot;,<br>        python_model=MultiModelPyfunc(),<br>        artifacts={<br>            &quot;models_encoded&quot;: &quot;/dbfs/tmp/ap/artifact.json&quot;<br>        },<br>    )<br>    print(&quot;Your Run ID is: &quot;, run.info.run_id)<br><br>    mv = mlflow.register_model(<br>        f&quot;runs:/{run.info.run_id}/augmented-fct-model-custom&quot;, <br>        f&quot;{model_serving_name}&quot;<br>    )<br>    client.transition_model_version_stage(<br>        f&quot;{model_serving_name}&quot;,<br>        mv.version,<br>        &quot;Production&quot;,<br>        archive_existing_versions=True,<br>    )</pre><p>So this JSON file that we’ve passed under the MLFlow Artifact actually contains all our serialized binary models and corresponding key-value pairs for each individual model.</p><pre>import json <br>with open(f&#39;/dbfs/tmp/ap/artifact.json&#39;, &#39;w&#39;) as outfile:<br>    json.dump(dict_stores_models, outfile)<br># {&quot;299_1&quot;: &quot;gASV73EAAAAAAACMIFNlcnZpbmdGb3JlY2FzdGluZy53cm.....&quot;,}</pre><p>Now let’s go further and understand how to actually this file will be used during the scoring process:</p><ol><li>When Serving Endpoint is created it loads the MLFlow artifact into memory, so the context of the model is loaded once</li><li>When we evoke the model(call an API) the model takes the input and treats it row by row (we are going to see in Part 4 how to pass the dataset per row)</li><li>Since we require to score more than 1 model at a time, we need to find a way how to get this model from the uploaded to the memory context.</li></ol><pre><br>class MultiModelPyfunc(mlflow.pyfunc.PythonModel):<br>    &quot;&quot;&quot;<br>    ids_store (list, str) ::<br>      id&#39;s that correponds to keys associated to the artifacts dict<br>      keys will be used to withdraw object that correpond to a particular model<br>      {id_name}: model_object_per_id<br><br>    &quot;&quot;&quot;<br><br>    def __init__(self):<br>        super().__init__()<br>        <br>    def load_context(self, context):<br>        # Get Dictionary with your artifacts for all models here<br>        json_load = context.artifacts[&quot;models_encoded&quot;]<br>        with open(json_load) as json_file:<br>            data = json.load(json_file)<br>        self.models_context_dict = data<br><br>    ## An example yif you need to add a data processing on the fly to your data<br>    def process_input(self, raw_input):<br>        pass<br><br>    def select_model(self, model_input):<br>        if not isinstance(model_input, pd.DataFrame):<br>            raise ValueError(&quot;Sample model requires Dataframe inputs&quot;)<br>        locale_id = f&#39;{model_input[&quot;Store&quot;].iloc[0]}_{model_input[&quot;Horizon&quot;].iloc[0]}&#39;  # getting the model id from the Store name<br>        return locale_id<br><br>    def predict(self, context, raw_input):<br>        &quot;&quot;&quot;<br>        expecting the input to be in array<br>        &quot;&quot;&quot;<br>        # We are checking if this is a Pandas.DF, <br>        # because this is the output from the Serving API into the Predict<br>        if type(raw_input) is type(pd.DataFrame()):<br>            raw_input = eval(raw_input.values[0][0])<br><br>        # We are here creating a DF for an expected income<br>        # if you want to retrain the model on the fly<br>        # place this under Try and add another one on except side<br>        model_input = pd.DataFrame(<br>            raw_input, columns=[&quot;Store&quot;, &quot;Horizon&quot;, &quot;Date&quot;, &quot;StateHoliday&quot;, &quot;SchoolHoliday&quot;]<br>        )<br>        model_input[&quot;StateHoliday&quot;] = model_input[&quot;StateHoliday&quot;].astype(&quot;bool&quot;)<br>        model_input[&quot;SchoolHoliday&quot;] = model_input[&quot;SchoolHoliday&quot;].astype(&quot;bool&quot;)<br>        model_input.sort_values(&quot;Date&quot;, inplace=True)<br><br>        try:<br>            selected_store = self.select_model(model_input)<br>            print(f&quot;Selected model {selected_store}&quot;)<br>            model_context = self.models_context_dict[str(selected_store)]<br>            model = pickle.loads(urlsafe_b64decode(model_context.encode(&quot;utf-8&quot;)))<br>            return model.predict(None, model_input)<br>        except:<br>            # Here you can fit your model, add an assert with corresponding amount of columns<br>            # pay attention your columns will not be the same as for Predict method<br>            return f&quot;This ID was not yet pre-trained, you input is {raw_input}, with a type {type(raw_input)}&quot;</pre><p>As you’ve seen above I am leveraging MLFlow context for this, this will load our object once into memory and keep it there while the Endpoint is running.</p><h3>Part 4 — Create your serving endpoints</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6q0--oqPTYIHFvPDAylodg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@alevisionco?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Alev Takil</a> on <a href="https://unsplash.com/photos/3syTDiVAc7w?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>As has been mentioned in Part 3, model serving would treat your input data by default row by row(as expected because it’s a service for low latency response), let’s have a look at the example of the input data expected by Databricks Model Serving(one of the options):</p><pre>  {<br>    &quot;dataframe_split&quot;: [{<br>      &quot;index&quot;: [0, 1],<br>      &quot;columns&quot;: [&quot;sepal length (cm)&quot;, &quot;sepal width (cm)&quot;, &quot;petal length (cm)&quot;, &quot;petal width (cm)&quot;],<br>      &quot;data&quot;: [[5.1, 3.5, 1.4, 0.2], [4.9, 3.0, 1.4, 0.2]]<br>    }]<br>  }</pre><p>But for a forecasting case, we require the dataset as a whole not row by row! For this, we require to be more “artistic” in preparing the input format for the Endpoint. This can be a tricky one, so let’s have a look at how I dealt with it:</p><pre># You would require to Serialize your Dataset into 1 row. <br># For this I&#39;ve created this small NumpyEncoder class.<br># Please note that you would pass this serializer under the Serving Function <br># when the API is evoked. Below it&#39;s a demonstration.<br>import json <br>class NumpyEncoder(json.JSONEncoder):<br>    def default(self, obj):<br>        if isinstance(obj, np.ndarray):<br>            return obj.tolist()<br>        return json.JSONEncoder.default(self, obj)<br><br>json_dump = json.dumps(input_df.values, cls=NumpyEncoder) <br> <br>ds_dict_serving = {<br>  &quot;dataframe_split&quot;: {<br>    &quot;index&quot;:[0], # [0,1,2] each index corresponds to a DF <br>    &quot;columns&quot; : [&quot;input_data&quot;], <br>    &quot;data&quot;:  [json_dump]<br>    }<br>}</pre><p>And this is it, when the model has been serialized and registered under the Model Serving component, you just need to create an Endpoint on Databricks Lakehouse (by using UI or API).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VsJXu5SNt5ywBwnKanTWvA.png" /><figcaption>I created my Endpoint via Databricks UI, which took me 3 clicks.</figcaption></figure><p>You can evoke the Endpoint like this (this is a very simple example, you can modify it accordingly):</p><pre>def score_model(ds_dict, token, url):<br>  headers = {&#39;Authorization&#39;: f&#39;Bearer {token}&#39;, &#39;Content-Type&#39;: &#39;application/json&#39;}<br>  ds_dict_testing = json.dumps(ds_dict, allow_nan=True)<br>  response = requests.request(method=&#39;POST&#39;, headers=headers, url=url, data=ds_dict)<br>  if response.status_code != 200:<br>    raise Exception(f&#39;Request failed with status {response.status_code}, {response.text}&#39;)<br>  return response.json()<br><br>score_model(ds_dict_serving, token, url)</pre><p>Later you would probably like to re-parse the output format since it’s going to be by default in a dictionary format.</p><p>Something that I would like to pay your attention to, is if you want to run the same model in a batch or streaming fashion you do not have to change anything.</p><p>Let’s have a look at how I scored my WrappedModel in batch(you just simply need to load it from your MLFlow Experiment or your MLFLow Model Registry):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o0gKCT7UT9pjFJYNcvu8uA.png" /></figure><p>Hope you like the content and this will help you in serving your many models under a single Endpoint. A small note before you move into the repository with the code example. Take into account that there are limitations and also added overheads when you are using a single Endpoint concept. Be aware of the concurrency(the number of requests you can execute concurrently, this can be adjusted accordingly to your Endpoint Size) and the size of your object that was uploaded into memory(if the object is big, scaling by default to 0 is strongly not recommended). The choice of the amount of Endpoints depends on the use case you&#39;re currently implementing(in simple words if you are doing fraud detection and forecasting under the same endpoint will definitely lead to further issues, for example on the model and data quality monitoring), so try to be realistic.</p><p>If you would like to see the full code you can check it on <a href="https://github.com/AnastasiaProkaieva/mmf_dbx_scaling">my GitHub page</a>.</p><p>I would like to thank Artem Sheiko, Michael Shtelma, and Jose Alfonso for reviewing the content.</p><h4>Things to read</h4><ul><li><a href="https://www.databricks.com/blog/2023/03/07/announcing-general-availability-databricks-model-serving.html">Announcing General Availability of Databricks Model Serving</a></li><li><a href="https://docs.databricks.com/machine-learning/model-serving/index.html">Model serving with Databricks</a></li><li><a href="https://mlflow.org/docs/latest/python_api/mlflow.pyfunc.html?highlight=load_context#mlflow.pyfunc.PythonModel.load_context">mlflow.pyfunc</a></li><li><a href="https://mlflow.org/docs/latest/models.html#model-customization">MLflow Models</a></li><li><a href="https://www.databricks.com/blog/2020/05/20/new-pandas-udfs-and-python-type-hints-in-the-upcoming-release-of-apache-spark-3-0.html">How Python type hints simplify Pandas UDFs in Apache Spark 3.0</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=80d5e3f32943" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why anyone should know Delta Lake if you work with Data?]]></title>
            <link>https://medium.com/@anastasia.prokaieva/why-anyone-should-know-delta-lake-if-you-work-with-data-b8c1e3636d60?source=rss-21aed4db6339------2</link>
            <guid isPermaLink="false">https://medium.com/p/b8c1e3636d60</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[delta-lake]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Anastasia Prokaieva]]></dc:creator>
            <pubDate>Tue, 04 Oct 2022 09:44:48 GMT</pubDate>
            <atom:updated>2023-05-24T11:21:13.976Z</atom:updated>
            <content:encoded><![CDATA[<h3>Why anyone should use Delta Lake while working with Data?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tAvEJL9KRdcTxTJ9caPWnw.png" /><figcaption>Welcome to your Lakehouse!</figcaption></figure><p>Data Science is all about data. If you feed your model with garbage, it will provide you with the same type of output — garbage.</p><blockquote>GIGO (garbage in, garbage out) is a concept common to computer science and mathematics: the quality of output is determined by the quality of the input.</blockquote><p>It becomes critical to anyone who works with data being able to keep track of data lineage and to “time travel” if necessary. Nowadays we hear more and more of a concept of a <a href="https://www.featurestore.org/what-is-a-feature-store#:~:text=The%20Feature%20Store%20is%20where,that%20have%20a%20trained%20model).">Feature Store</a>, new type of modules or even platforms that helps people to create a <strong>direct connection</strong> between their <strong>data</strong>, <strong>models</strong> and <strong>business</strong>. At the end of the day it does not matter how great your code is, if your business is not growing. If you want to know more about the concept of a Feature Store, I would recommend to check this <a href="https://mlopshowto.com/what-are-feature-stores-and-why-are-they-critical-for-scaling-machine-learning-94e14afec81d">post</a> that was written by <a href="https://medium.com/u/83470b348bae">Rafael Pierre</a>.</p><p>But let’s come back for a moment to the first principles and main brick of this story to become a success — it’s your data and it’s lineage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kg9E_o5yiFdP8fR4NND0uQ.png" /><figcaption>Your data should flow and not being siloed.</figcaption></figure><p>Let’s walk through a simple working week of a data team: you would get to your desk grab some beverages and dive deep into your code. Often depending on your job title, you may do some ETL with PySpark, SQL, Python or even Scala in order to ingest your data from all the sources; then you move to your feature preparation or what is often called feature engineering, when they are done you would often save them somewhere:</p><blockquote>this maybe a database; a simple blob storage with a csv or even a parquet file! or you may just continue to work and keep them in your memory cached (don’t do this please).</blockquote><p>Depending whether it’s you who is creating a model or not, the model will be built based on those features you’ve just created.</p><p>Now let’s travel forward in time, imagine you and your colleagues that were working with you on this project are absent and your data owner urgently requires to know which data have been used for this model (because maybe they identified that freshly arriving data were renormalized recently, and they don’t know which features you and your colleagues have used to train your model).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ugK3S86cyp5HoVGvE0qISQ.jpeg" /><figcaption>Oops</figcaption></figure><p>Because you are not there, but your data team have access to your code, when connected, they realized that you have multiple files inside your production code folder. Those files have different time_stamps and versioning but none of those are corresponding to what you have used in your code.</p><p>Often time, files in production looks like this(unfortunately, but trust me I’ve seen it all around):</p><blockquote>data_prod_2018:01:12:00:00:00_dataProc_v3.csv</blockquote><p>Then your data owner decides to check your ML teammate code, to find out which data they actually fed to the model that are running in the production, they click run all and hit an error:</p><blockquote>exception was raised “please verify your file path, it seems to not exist”</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r-x-xUA1oYL1Xc8mHaivmQ.jpeg" /><figcaption>Your data is not even there! We are on fire…</figcaption></figure><p>Let’s imagine another scenario, when your production code was running just fine and your team was using some online SQL database where your data were ingested until an external data provider changed their schema names and your code is not functioning anymore and you need to go and urgently rerun plenty of tests and modify your schema input manually.</p><p>This gives you an idea why this is so important to have a direct lineage between your data source, your model input and your model output. This is where a direct integration of a Databricks’ Feature Store with MlFlow helps a lot! But what is under the hood of this magical Feature Store from Databricks? It’s just a <a href="https://docs.delta.io/latest/quick-start.html">Delta Lake</a>!</p><h4>Delta Lake Tables what are they?</h4><p>Here it comes to Delta ! Initially created by <a href="https://www.databricks.com/">Databricks</a> engineering team now is completely open source(from summer 2022). Delta Lake is an open-source storage layer that sits on top of your data lake(cloud storage). It is built for distributed computing and 100% compatible with Apache Spark, so you can easily convert your existing data tables from whatever format they are currently stored in (CSV, Parquet, etc.) and save them as a Bronze table in Delta Lake format using your favorite <a href="https://www.databricks.com/glossary/spark-api">Spark APIs</a>.</p><blockquote><a href="https://delta.io/">Delta Lake</a> is an <a href="https://github.com/delta-io/delta">open source project</a> that enables building a <a href="https://databricks.com/blog/2020/01/30/what-is-a-data-lakehouse.html">Lakehouse architecture</a> on top of <a href="https://databricks.com/discover/data-lakes/introduction">data lakes</a>. Delta Lake provides <a href="https://docs.delta.io/latest/concurrency-control.html">ACID transactions</a>, scalable metadata handling, and unifies <a href="https://docs.delta.io/latest/delta-streaming.html">streaming</a> and <a href="https://docs.delta.io/latest/delta-batch.html">batch</a> data processing on top of existing data lakes, such as S3, ADLS, GCS, and HDFS.</blockquote><p>Main principle of Delta Lake is that is allows you to work with your data back-forward, avoiding creating a so called <a href="https://www.databricks.com/glossary/lambda-architecture">lambda architecture</a>. A common architecture of any data project uses tables that correspond to different quality levels in the data engineering pipeline, progressively adding structure to the data: data ingestion (“Bronze” tables), transformation/feature engineering (“Silver” tables), and machine learning training or prediction (“Gold” tables).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6t9hM5Ho5RM9YG9GuOPW8A.png" /><figcaption>Avoid lambda architecture with Delta Lake Medallion Architecture!</figcaption></figure><p>It’s very easy to start with, and I’ll walk you through some life changing features. Features to know (from my day to day job as an ML Engineer):</p><ul><li><a href="https://docs.delta.io/latest/concurrency-control.html">ACID transactions</a> on Spark: Serializable isolation levels ensure that readers never see inconsistent data. Imagine you and your colleague that are have access to the same table and call write on it!</li><li>Scalable metadata handling: Leverages Spark distributed processing power to handle all the metadata for petabyte-scale tables with billions of files at ease. Delta Lake uses so called delta_log in order to track changes that were made.</li><li><a href="https://docs.delta.io/latest/delta-streaming.html">Streaming</a> and <a href="https://docs.delta.io/latest/delta-batch.html">batch</a> unification: A table in Delta Lake is a batch table as well as a streaming source and sink. Streaming data ingest, batch historic backfill, interactive queries all just work out of the box. Imagine you need to run your model on a file trigger, what do you do, run your job constantly and hope that file has landed in between those ? Of course not, you would use <a href="https://towardsdatascience.com/using-databricks-autoloader-to-support-event-driven-data-ingestion-e833f4306d55">AutoLoader with Spark Streaming</a> while your files are arriving and a model is running on top of them, you can continue to train your models.</li><li><a href="https://www.databricks.com/blog/2019/09/24/diving-into-delta-lake-schema-enforcement-evolution.html">Schema enforcement</a> and evolution: Automatically handles schema variations to prevent insertion of bad records during ingestion. Not only this will not permit bad files to pass, but it also allows you to evolve your schema if needed without fully rewriting your tables.</li><li><a href="https://docs.delta.io/latest/delta-batch.html#-deltatimetravel">Time travel</a> a.k.a data versioning: Data versioning enables rollbacks, full historical audit trails, and reproducible machine learning experiments. My favourite example, imagine today you applied some transformation to your data, but you are not sure whether the previous version was better or not, instead of going back and running your code again you just can access a particular version of your data (and your model knows which one you used so the comparison becoming super easy)!</li><li><a href="https://docs.delta.io/latest/delta-update.html#-delta-merge">Upserts</a> and <a href="https://docs.delta.io/latest/delta-update.html#-delta-delete">deletes</a>: Supports <a href="https://www.databricks.com/blog/2020/09/29/diving-into-delta-lake-dml-internals-update-delete-merge.html">merge, update and delete</a> operations to enable complex use cases like change-data-capture(CDC), slowly-changing-dimension (SCD) operations, streaming upserts, and so on. Simply update your freshly arriving data!</li></ul><p>Other features you should know in addition to everything stated above :</p><ul><li>Vacuum: Cleans up the untracked and redundant files to limit storage costs. It’s been a while you don’t use your files anymore? Set up how long are you ready to keep them inside your storage (please keep in mind that here I relate to a Standard storage type, not a Glacier type)!</li><li>Optimize: Partition data to improve query performance, (with and without ZORDER) performs compaction, where possible to 1 GB file each and improves data skipping. Are you always running your aggregations or models over regional / seasonal data — partition over them, and speed up data reading.</li><li>ZORDER: For Higher Cardinationlity columns, is a technique to colocate related information in the same set of files. Further improves data skipping. Allows for sequentially efficient reads from cloud storage, decreasing total time to read data from storage into memory.</li><li>Clones — deep and shallow: Used for data archiving, short-term experiments, data sharing and disaster recovery. Stop copying your data all around the place, use clones!</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/510/1*C5ydq5B6RQ_RxZ875PU5rw.jpeg" /><figcaption>Clones all around!</figcaption></figure><p>If you want to understand how exactly Delta Lake works under the hood I highly recommend to check this <a href="https://www.databricks.com/blog/2019/08/21/diving-into-delta-lake-unpacking-the-transaction-log.html">Deep Dive into Delta Log</a>.</p><p>All code-examples could be found on the <a href="https://docs.delta.io/latest/index.html">main page of the Delta Lake</a>. A very <a href="https://www.databricks.com/p/webinar/delta-lake-the-foundation-of-your-lakehouse">great webinar</a> exists on demand from Databricks. I mostly run Delta Lake with it’s Python or SQL API.</p><p>My personal favourite <a href="https://www.databricks.com/blog/2019/08/14/productionizing-machine-learning-with-delta-lake.html">blog</a> that walks you through Bronze — Silver — Gold medallion architecture and where Delta Lake sits strong and how it simply can be used together with MlFlow.</p><h4>Delta Lake on Databricks</h4><p>Is there any difference between using Delta Lake open source on Databricks or anywhere else?</p><p>The difference will be mainly coming from the fact that on Databricks Delta Lake and Apache Spark are directly integrated into the Lakehouse platform. Databricks is also preconfiguring best perks for you as a user, and then you don’t have to even think about what is a <a href="https://www.databricks.com/blog/2021/09/08/announcing-public-preview-of-low-shuffle-merge.html">low-shuffle merge</a> or <a href="https://www.databricks.com/blog/2020/05/29/adaptive-query-execution-speeding-up-spark-sql-at-runtime.html">AQE</a> or how to set up your Spark Config. MlFlow is directly and seamlessly integrated with Databricks Feature Store and the platform itself— your data and model lineage in a single place. You can add an additional governance layer to your data with the <a href="https://www.databricks.com/product/unity-catalog">Unity Catalog</a> and share your data with the <a href="https://www.databricks.com/product/delta-sharing">Delta Sharing</a>!</p><p>In addition Databricks recently announced a new powerful computational engine — <a href="https://www-cs.stanford.edu/~matei/papers/2022/sigmod_photon.pdf">Photon</a> - written from the ground up in C++, Photon takes advantage of modern hardware for faster queries, providing up to 12x better price/performance compared to other cloud data warehouses — all natively on your data lake.</p><p>Apart of fast computations on the platform your data stays at your account level and are not blocked into proprietary data formats(compare to the most of other “data” platforms on the market), then you can simply use Databricks Lakehouse platform to process your SQL queries or to train an image recognition deep learning model — all in one platform with your own data.</p><h4>Coming Next</h4><p>At our next stop we will work on a real hands-on example of the Databricks Feature Store powered by Delta and integrated directly with MlFlow. Hope you’ll enjoy the read and found yourself within a few stories of mine.</p><h4><strong>Resources</strong></h4><ol><li>Great book : “Simplifying Data Engineering and Analytics with Delta” by <a href="https://www.wob.com/en-gb/books/author/anindita-mahapatra">Anindita Mahapatra</a></li><li><a href="https://delta.io/">https://delta.io/</a></li><li><a href="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html">https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html</a></li><li><a href="https://www.mlflow.org/docs/latest/index.html">https://www.mlflow.org/docs/latest/index.html</a></li><li><a href="https://www.databricks.com/">https://www.databricks.com/</a></li></ol><h3>Other stories you might like</h3><ul><li><a href="https://mlopshowto.com/how-lakehouse-powers-nlp-for-customer-service-analytics-in-insurance-e5bd9d860c5e">How Lakehouse powers NLP for Customer Service Analytics in Insurance</a></li><li><a href="https://mlopshowto.com/keeping-your-machine-learning-models-on-the-right-track-getting-started-with-mlflow-part-2-bbc980a1f8dc">Keeping Your Machine Learning Models on the Right Track: Getting Started with MLflow, Part 2</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b8c1e3636d60" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>