<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Joshua Phuong Le on Medium]]></title>
        <description><![CDATA[Stories by Joshua Phuong Le on Medium]]></description>
        <link>https://medium.com/@joshuale?source=rss-d91fba39f365------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*qmmdROW7E7m4T4Gr</url>
            <title>Stories by Joshua Phuong Le on Medium</title>
            <link>https://medium.com/@joshuale?source=rss-d91fba39f365------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 02:59:33 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@joshuale/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[“AI is eating my lunch”: The challenge of junior level developers and the risk of organizations]]></title>
            <link>https://medium.com/@joshuale/ai-is-eating-my-lunch-the-challenge-of-junior-level-developers-and-the-risk-of-organizations-dde10cb1853a?source=rss-d91fba39f365------2</link>
            <guid isPermaLink="false">https://medium.com/p/dde10cb1853a</guid>
            <category><![CDATA[generative-ai-use-cases]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Joshua Phuong Le]]></dc:creator>
            <pubDate>Sat, 20 Jun 2026 04:30:01 GMT</pubDate>
            <atom:updated>2026-06-20T04:30:01.953Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R1IqqSXSS0h9U70ZIOCguA.jpeg" /><figcaption>AI-generated by Grok</figcaption></figure><p>Coding copilots and agentic dev tools are getting good — fast. And the obvious move for many teams (or at least the CEOs) has become:</p><ul><li>fewer junior hires</li><li>more output from fewer people</li><li>seniors spend time reviewing/approving AI-generated code</li></ul><p>Sounds efficient. There’s no denying of the real gain in time efficiency, and time saving is real money saving and improvement on company bottom-line.</p><p>But there’s a hidden trade: <strong>short-term velocity for long-term capability.</strong></p><h3>⚙️ The uncomfortable dynamic</h3><p>Traditionally, juniors do a lot of the “typing work” (repetitions), while building real-world instincts:</p><ul><li>reading messy codebases</li><li>debugging production issues</li><li>learning patterns through repetition</li><li>getting corrected in reviews</li><li>absorbing “why we do it this way” through human back-and-forth</li></ul><p>If AI takes the typing… and seniors only <em>approve</em>…</p><p><strong>where do future seniors come from?</strong></p><h3>⚠️ The long-term risks most teams are underestimating</h3><h3>1) Your talent pipeline quietly collapses</h3><p>People leave (they always do). When experienced engineers churn, you normally have juniors → mids to step up.</p><p>If you stop feeding that pipeline, you eventually hit a cliff:</p><ul><li>not enough internal replacements</li><li>hiring externally is slower and loses context</li><li>institutional knowledge leaks permanently</li></ul><h3>2) Review becomes rubber-stamping (without you realizing)</h3><p>Reviewing AI code isn’t the same as reviewing a peer:</p><ul><li>there’s less debate and pushback</li><li>the assistant optimizes for “pleasing” you, not challenging assumptions</li><li>it’s easier to accept plausible-looking code than to interrogate it deeply</li></ul><p>Over time, seniors can become <em>approvers</em>, not <em>builders</em>.</p><h3>3) Troubleshooting quality degrades</h3><p>When incidents happen, you don’t need “code output.”</p><p>You need people who can:</p><ul><li>form hypotheses</li><li>reason about system behavior</li><li>navigate weird edge cases</li><li>understand historical failures and tradeoffs</li></ul><p>Those skills come from years of reps. <strong>AI can’t replace the reps if nobody gets to do them.</strong></p><h3>4) Everyone assumes the “ideal scenario”</h3><p>Many leaders implicitly assume the AI agents:</p><ul><li>know your coding standards</li><li>understand your multi-repo architecture</li><li>remember historical failures and lessons</li><li>operate under a robust access-control model (same rigor as humans)</li></ul><p>Those are massive <em>IFs</em>. Solving even one is real engineering work.</p><p>These are common blindsides for non-technical leaders, at no fault of their own. The technical leaders should act as the balancing force to assess the practicality and hidden risks of certain visions and directions.</p><h3>🧭 Risk ≠ doom. But it requires intentional management.</h3><p>In general, I’m not arguing against using AI agents, neither am I the first one to jump on new bandwagons. In my opinion, if you want AI leverage without hollowing out your engineering org, you have to design for:</p><ul><li>how juniors still get real reps (not just “prompting”)</li><li>how knowledge is captured and transferred (human↔human, human↔AI)</li><li>how reviewers stay technically engaged (not just approving diffs)</li><li>what parts of engineering you refuse to outsource</li><li>for outsourced tasks, how do you still ensure technical oversight and have a disaster recovery playbook.</li></ul><p>These are important questions that real human managers, business or technical alike, should ponder over, take suitable actions that balance both short-term and long-term benefits and costs.</p><p>How are you handling this in your team — are you hiring fewer juniors, redefining the junior role, or investing in a different training path?</p><p><em>*Disclaimer: this article is written with the help of an AI assistant for style purposes. All content and opinions belong to the author.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dde10cb1853a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Mental Models for AI Workflows, Agents & Agentic Systems — and why the choice is never free.]]></title>
            <link>https://medium.com/@joshuale/mental-models-for-ai-workflows-agents-agentic-systems-and-why-the-choice-is-never-free-8d12ac77e944?source=rss-d91fba39f365------2</link>
            <guid isPermaLink="false">https://medium.com/p/8d12ac77e944</guid>
            <dc:creator><![CDATA[Joshua Phuong Le]]></dc:creator>
            <pubDate>Mon, 27 Apr 2026 02:10:41 GMT</pubDate>
            <atom:updated>2026-04-27T02:10:41.238Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-53JMuM4qxKDqlgZe6mQuA.jpeg" /><figcaption>AI-generated by Grok</figcaption></figure><p>Everyone’s building “agents” right now. But without a clear mental model of what you’re actually building, you’ll under-engineer the simple things and sleepwalk into the complex ones.</p><p>Here’s how I think about it. 👇</p><h3>🗺️ The Taxonomy</h3><ul><li>⚡ AI Workflow — predefined, explicit execution paths. Predictable, testable, debuggable. Your first instinct for structured problems.</li><li>🤖 AI Agent — autonomous, specialized, capable of dynamic multi-turn decision-making. More powerful, but now you’ve taken on session state, streaming, and non-determinism.</li><li>🕸️ Agentic AI System — a coordinator that decomposes complex tasks and routes them to specialized agents. Power multiplied — and so is the complexity.</li><li>⚠️ Hybrid — this is where the lines blur. A workflow with agent nodes. An agent that calls a workflow as a tool. An agentic system mixing both. These exist in production and they’re legitimate — but they demand you know <em>exactly</em> what guarantees each component provides.</li></ul><h3>📐 The Choice Has Engineering Consequences</h3><p>It’s not just architecture philosophy — it changes your entire stack.</p><p>🔌 A workflow maps cleanly to a REST API with fixed input/output schemas. 📡 An agent needs long-running sessions and streaming — conventional REST becomes a liability, not a solution.</p><p>And complexity scales <em>non-linearly</em>. A few examples:</p><p>🔍 Adding RAG to an agent isn’t just plugging in a vector DB. You inherit:</p><ul><li>Document processing &amp; vectorization pipelines</li><li>Separate evals for retrieval <em>and</em> generation</li><li>Closed-loop feedback collection (human or LLM judges)</li><li>Failure mode identification and recovery playbooks</li></ul><p>🔧 When agent tools proliferate, they outgrow project repos and need a shared platform (MCP servers, internal tool registries). Now you’re asking:</p><ul><li>Who owns and maintains these tools?</li><li>Which agents can access what, in which environments?</li><li>Who pays for them across projects?</li></ul><p>🛡️ Guardrails for user-facing systems aren’t a feature — they’re a discipline. Building, testing, and <em>maintaining</em> them as your system evolves is a continuous engineering investment.</p><h3>🎯 3 Lessons That Don’t Change</h3><p>1️⃣ Choose the right paradigm — then own the engineering cost that comes with it. There is no free lunch. The simplest system that solves the business problem is the right system.</p><p>2️⃣ Start small, evaluate early, add complexity only when it unlocks something critical. Agentic complexity added prematurely is technical debt with an unpredictable interest rate.</p><p>3️⃣ A working POC (or a salesperson’s pitch) is not a production system. The gap between “it works in a demo” and “it meets enterprise security, safety, and governance standards” is where most projects actually live — and where most effort is actually spent.</p><p>━━━━━━━━━━━━━━━</p><p>The tooling is maturing fast. The engineering judgment required to use it well isn’t automatic.</p><p>💬 Where have you seen teams underestimate the jump from workflow → agent → agentic system?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8d12ac77e944" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Managed MLflow Service (for Free)]]></title>
            <link>https://medium.com/mitb-for-all/using-managed-mlflow-service-for-free-24fab870c75a?source=rss-d91fba39f365------2</link>
            <guid isPermaLink="false">https://medium.com/p/24fab870c75a</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[mlflow]]></category>
            <dc:creator><![CDATA[Joshua Phuong Le]]></dc:creator>
            <pubDate>Sat, 03 Jan 2026 12:47:10 GMT</pubDate>
            <atom:updated>2026-01-08T08:22:19.506Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Vt0YKxp2gx1j3lmN" /><figcaption>Photo by <a href="https://unsplash.com/@glenncarstenspeters?utm_source=medium&amp;utm_medium=referral">Glenn Carstens-Peters</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>1. Motivation</h3><p>For a long time, I’m an advocate for MLflow to manage your ML model lifecycle and development work due to its ease of use and constant updates to accommodate the ever-evolving needs — MLflow 3+ being the most recent major upgrade to better work with GenAI.</p><p>You can definitely set up your own MLflow service (a bit of that in the next section), but if you just want to start using a managed service that is both free and available for all of your projects, regardless of your development environment, then using MLflow on Databricks Free Edition ticks all these boxes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YrOf5osY4B4904v66_8FEA.png" /><figcaption>Source: author</figcaption></figure><p>While using MLflow by directly coding on Databricks happens automatically (and there are many tutorials elsewhere to show you), to many developers who are used to and most efficient with working with the local code editor, coding on Databricks does not sound attractive. This is because suddenly you have to leave the comfort and convenience of the local code editor with all your efficient navigation, favorite add-ons, and then deal with new concepts such as compute clusters, catalogs, a different way to manage dependencies (no conda/venv there). To my experience, this full transition is probably one of the biggest hurdles to overcome when speaking with many data scientists.</p><p>Thus, a hybrid approach illustrated below with just using the remote MLflow service while keeping everything else locally is much more appealing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7eXovXhFT5obxAgk5fqgKQ.png" /></figure><p>If you read up Databricks documentation, there are several methods to get this working. But the information can be overwhelming as it tries to explain many scenarios and give you many options. This blog is only focusing on a single case: <strong>you as an individual during development phase, logs ML experiments to a remote MLflow server from your PC with OAuth Practice (avoid using Personal Access Tokens — PAT*).</strong></p><blockquote>PAT is strongly discouraged, or sometimes outright disabled in the enterprise environment due to many reasons (you can Google and read up more). Thus, I want to focus on a method that will work on both your personal and organizational scenarios.</blockquote><p>Overall, we will perform 4steps below. Steps 1–3 are done only once.</p><ul><li>Step 1: Create a Databricks Free Edition account.</li><li>Step 2: Install the Databricks CLI</li><li>Step 3: Setup User-to-Machine (U2M) OAuth</li><li>Step 4: we will test the connection with the MLflow Python SDK from the local PC by logging a sample experiment run.</li></ul><h3>2. A Gentle Introduction to the MLflow Service</h3><p><em>This section is completely optional if you want to know a bit more about the MLflow backend. You can skip to section III to go straight to the setup method.</em></p><p>As a data scientist, carrying out machine learning model iterative development is the bread-and-butter requirement, be it in our formal job requirement or as our side projects. Having a tool to systematically organize and log our experiments becomes a necessity, not only during development but also in model deployment to inference endpoints or batch jobs.</p><p>A fully functional MLflow service will have 4 components:</p><ul><li>An artifact store for your model weights, logged images,</li><li>A backend store for structured information metadata linking everything such as which runs belong to which experiments, traces, etc.</li><li>A tracking server (FastAPI behind the scenes) with an UI for interacting with everything by clicking through or REST APIs.</li><li>Finally, the SDKs for users to work with everything above.</li></ul><p>MLflow is available as (1) an open-source software for you to install and manage, or (2) as a managed service.</p><h4>2.1. Self-Hosting</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*naoCLqQScQaxMvjG.png" /><figcaption>Different Modes of Self-Hosting (Source: MLflow)</figcaption></figure><p>If you just want to self-host everything, setting up the MLflow service for your local development can get from extremely easy with just a pip install where everything is stored in your local PC main file system, to tricker with a containerized approach that comes with all the services linked together. Check out the documentation here for more information (<a href="https://mlflow.org/docs/latest/self-hosting/architecture/overview/">Architecture Overview | MLflow</a>).</p><p><strong>The pros</strong>: you control everything.</p><p><strong>The Cons</strong>: you need to setup and manage everything. While setting up something for yourself can be easy, setting up a collaborative service for your team needs IT expertise that goes beyond what data scientists are good at.</p><h4>2.2. Managed Service</h4><p>If you prefer a managed service, MLflow comes pre-built on major ML platform providers such as Databricks, Azure ML, or AWS SageMaker.</p><p><strong>The Pros:</strong> You can leave the setup and management of service to the provider and focus on the development work.</p><p><strong>The Cons:</strong> for most providers, you need to sign up for a non-free/non-trial account for long-term usage.</p><h3>3. Using a Managed MLflow for Free with Databricks</h3><p>Recently, during the 2025 World Tour event, Databricks introduced a new type of account for their platform — the <strong>Free Edition </strong>(FE), which replaces their legacy Community Edition (CE).</p><p>With CE, while users could use the MLflow Tracking API to log parameters, metrics, and artifacts directly, key components like the <strong>Model Registry</strong>, <strong>Model Serving</strong> (deployment) were not supported.</p><p>With FE, you can now use all crucial features available in the paid edition above, including the Unity Catalog. Of course there will be some restrictions, especially on powerful compute/serving features, but for the intent and purposes of starting to use MLflow, and bringing it through the deployment process, you will not be much handicapped.</p><p>This blog will only focus on the MLflow connection setup to get you started. I can write more about other Databricks aspects in future blogs.</p><h4>Step 1: Sign Up for Databricks Free Edition</h4><p>If you don’t have one yet, you can create a new Databricks FE account easily without any payment information. Just go to its website and follow the instructions to sign up with your email. For those who enjoy a paid edition, by yourself or by your organization, everything will still apply.</p><p><a href="https://www.databricks.com/learn/free-edition">Free Edition | Replacing Databricks Community Edition</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dmysNbPsjA-FStWXE9wNhw.png" /></figure><p>Once signed up and logged in with your email, you will see this landing page. Notice 2 things:</p><ul><li>The URL of your workspace — this is the “workspace URL” to log in with the CLI later.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*Zjlcb1FLYsuGT8uTmzTtrg.png" /><figcaption>A sample host URL</figcaption></figure><ul><li>The AI/ML section on the left panel, you can see the Experiment subsection. This is the centralized location to see and collaborate on all MLflow experiments on the workspace.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GuGj6tXMamuetPwlBUuflw.png" /></figure><h4><strong>Step 2: Install Databricks CLI on Your Local Machine</strong></h4><p>Main reference: <a href="https://docs.databricks.com/aws/en/dev-tools/cli/install">Install or update the Databricks CLI | Databricks on AWS</a>. <br>For my setup (WSL2 on Windows), I used the following:</p><ul><li>First, install the unzip utility, which is not pre-installed with WSL2, by sudo apt install unzip</li><li>Then, install the CLI (with sudo) by</li></ul><pre>$ curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sudo sh</pre><p>If you need to install the CLI from the source zip file, do the following steps:</p><ul><li>Follow this guide to download the zip file for your system and extract it to a stable folder: <a href="https://docs.databricks.com/aws/en/dev-tools/cli/install#source-installation-for-linux-macos-and-windows">Install or update the Databricks CLI | Databricks on AWS</a></li><li>Then add the path of the folder above to your system:</li></ul><pre># PowerShell<br># Replace with the folder where you put databricks.exe<br>$folder = &quot;C:\Tools\databricks&quot;<br>[Environment]::SetEnvironmentVariable(<br>  &quot;Path&quot;,<br>  $env:Path + &quot;;&quot; + $folder,<br>  &quot;User&quot;<br>)<br><br># Start a new terminal, then verify:<br>databricks -v</pre><pre><br># <br># If you placed the binary in /usr/local/bin (recommended), you may not need this step.<br># For a user-local install in ~/bin:<br>mkdir -p ~/bin<br>mv /path/to/extracted/databricks ~/bin/<br>chmod +x ~/bin/databricks<br><br># Add to PATH (bash)<br>echo &#39;export PATH=&quot;$HOME/bin:$PATH&quot;&#39; &gt;&gt; ~/.bashrc &amp;&amp; source ~/.bashrc<br><br># Restart the shell and verify<br>databricks -v</pre><h4>Step 3: Create U2M Profile</h4><p>Main reference: <a href="https://docs.databricks.com/aws/en/dev-tools/cli/authentication#oauth-user-to-machine-u2m-authentication">Authentication for the Databricks CLI | Databricks on AWS</a></p><p>In our scenario, we will use:</p><ul><li>The user-to-machine (U2M) authentication during development work. The other machine-to-machine (M2M) authentication workflow is used during unattended processes such as CI/CD pipelines.</li><li>The workspace-level commands because the action of MLflow experiment logging is a workspace activity, not an account activity — the MLflow server is tied to a workspace.</li></ul><p>Now, with the CLI installed, you can run the following command — using the workspace URL in step 1.</p><pre>databricks auth login --host https://bla-bla.cloud.databricks.com</pre><p>CLI kicks off an OAuth user-to-machine (U2M) flow. When prompted with Databricks profile name by the CLI, type a name for your profile (e.g., joshuale-commonbelow). CLI will then open the workspace in your browser, or prompt you to do so with the authentication link. Follow the usual logging in process in the browser, once successful, this page will appear.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/404/1*e_Bc2fm9YRY7SUplZjW0Wg.png" /></figure><p>What happens is that the profile you created is now registered with your login credentials and stored securely. The OAuth process of requesting and exchanging a short-lived token will happen automatically behind the scenes whenever you interact with the remote Databricks resources as a user without repeatedly logging in (e.g., when using the MLflow SDK).</p><p>To check the profiles available from your PC and their respective authenticated workspace URLs, you can run the following command. As you can see in my example, I set up 1 joshuale-common profile for my Databricks FE workspace.</p><pre>~$ databricks auth profiles<br>Name             Host                                            Valid<br>joshuale-common  https://dbc-564fb500-5a75.cloud.databricks.com  YES</pre><h4>Step 4: Logging a MLflow experiment to the remote server</h4><p>Now in your local PC, install the mlflow[databricks] Python library from pip into your conda/virtual environment. At the point of writing this (Dec 2025), I’d recommend the version &gt;=3.1,&lt;3.5 for better stability with the Databricks UI.</p><pre>pip install mlflow[databricks]&gt;=3.1,&lt;3.5</pre><p>Finally, you can start logging a sample experiment as follows. Please read up on the mlflow Python APIs for further information.</p><pre>import mlflow<br><br><br>mlflow.set_tracking_uri(&quot;databricks://joshuale-common&quot;) # this profile already stores the remote workspace URL<br><br># 2) set the experiment you want to log to<br># this will create the experiment if it doesn&#39;t exist<br>mlflow.set_experiment(&quot;/Shared/your-shared-experiment&quot;) # a shared location<br># mlflow.set_experiment(&quot;/Users/&lt;your-user-email&gt;/path/to/your-personal-experiment&quot;) # your home page<br><br># 3) start a run and log stuff<br>with mlflow.start_run() as run:<br>    print(run.info.run_name)<br>    mlflow.log_param(&quot;p&quot;, 42)<br>    mlflow.log_param(&quot;model_type&quot;, &quot;demo&quot;)<br>    <br>    for i in range(5):<br>        mlflow.log_metric(&quot;metric_val&quot;, i * 2)<br><br>print(&quot;done&quot;)</pre><p>Note 2 things:</p><ul><li>I specify the tracking URI with my profile. MLflow will then authenticate via the credentials that the CLI has stored following the OAuth2 protocol — which you don’t have to worry about.</li><li>I set an experiment in the Shared/ location on the Databricks workspace. This is something specific to the Databricks plarform. You must always choose a location on the workspace that you have write-access to so that the Experiment objects can be saved. By default, you will have access to 2 places — (1) the “Shared” folder (accessible by everyone on the workspace) with the workspace path syntax /Shared/your-exp-name, and (2) your “Home” folder (accessible only by you and the admin — who is also yourself in our example if you created the workspace) with the path /Users/&lt;your-email&gt;/your-exp-name. For (2), you can create subfolders to store experiment objects and change the path accordingly.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qabSvJak6Nr2fSjm4uKO-A.png" /><figcaption>Locations on a workspace to find experiment objects</figcaption></figure><p>You can navigate to the AI/ML — Experiments section and see all experiments that were ever logged into the workspace and are accessible by you (you were the author, or someone shared the experiments with you…)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/838/1*dMqZBrvsVowHnqChwqE-aQ.png" /><figcaption>The centralzied experiment page</figcaption></figure><p>Once clicking the experiment in either of the places, you can go to the logged runs and see the usual metrics and parameters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aqWRBF2qiNRwYDQ5sm2Jrw.png" /><figcaption>The run information</figcaption></figure><h3>4. Conclusion</h3><p>This blog outlines the most straightforward method to start using a managed MLflow service for free with Databricks Free Edition, while ensuring secured connection, applicable in both personal and enterprise scenarios.</p><p>The underlying mechanism is also applicable to use other features of this platform from the comfort of your local IDE such as spark-enabled computes, Databricks Asset Bundles, Mosaic Model Serving, etc. I will cover some of these in future blogs.</p><p><em>Disclaimer: All opinions and interpretations are that of the writer, and not of MITB. I declare that I have full rights to use the contents published here, and nothing is plagiarized. I declare that this article is written by me and not with any generative AI tool such as ChatGPT. I declare that no data privacy policy is breached, and that any data associated with the contents here are obtained legitimately to the best of my knowledge. I agree not to make any changes without first seeking the editors’ approval. Any violations may lead to this article being retracted from the publication</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=24fab870c75a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mitb-for-all/using-managed-mlflow-service-for-free-24fab870c75a">Using Managed MLflow Service (for Free)</a> was originally published in <a href="https://medium.com/mitb-for-all">MITB For All</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Case of Databricks for Data-to-AI Platform — A Data Scientist’s Perspective]]></title>
            <link>https://medium.com/mitb-for-all/the-case-of-databricks-for-data-to-ai-platform-a-data-scientists-perspective-fe72fd0ac2d1?source=rss-d91fba39f365------2</link>
            <guid isPermaLink="false">https://medium.com/p/fe72fd0ac2d1</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Joshua Phuong Le]]></dc:creator>
            <pubDate>Thu, 22 May 2025 16:24:25 GMT</pubDate>
            <atom:updated>2025-05-23T12:41:03.568Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Case of Databricks for Data-to-AI Platform — Through A Data Scientist’s Lens</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kBxWuq46KJ_BicJn" /><figcaption>Photo by <a href="https://unsplash.com/@tavi004?utm_source=medium&amp;utm_medium=referral">Octavian Rosca</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>After spending 6 months with Databricks to design and implement our company MLOps processes together with other personas like data engineers and administrators, I have collected a list of observations that I hope can help some of you gain more clarity, especially those who are still consider your options.</p><p>The following diagram is my attempt to bucket commonly used Databricks features into verticals and layers according to my own understanding and data science / ML perspective (especially MLOps). These boundaries are not officially defined by Databricks and sometimes are not clear-cut, but they should help with the platform orientation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KpN9H4Ktt39sqGJhXUrLLw.png" /><figcaption>Databricks Overview from Data Science / ML Perspective</figcaption></figure><blockquote>Note: though the specific information is about Databricks offerings, the concepts are universal in all MLOps platforms. You can reuse them to evaluate and assess your options.</blockquote><p>While other platforms and “boutique” tools solve some problems along the code-data-model-serving-monitoring pathway, from my personal experience, Databricks offers a more complete suite as below. These cover a wide range of considerations ranging from backend (DE, DS) to serving, and the recently developed features for GenAI and front-end.</p><p>In the writeup below, I do a bit deeper to some key components in the diagram such as the DevOps — DataOps — ModelOps verticals and governance features.</p><p>The “formula” MLOps = DevOps + DataOps + ModelOps is again inspired by Databricks, which I believe is the right way to approach MLOps as any ML model is the direct result of data, and must be developed and maintained with DevOps principles. You can read up more in the documentation below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/0*4xQXl-uxbcFxP1GE.png" /><figcaption>Source: Databricks</figcaption></figure><p><a href="https://docs.databricks.com/aws/en/machine-learning/mlops/mlops-workflow">MLOps workflows on Databricks | Databricks Documentation</a></p><h3>I. Core Functions</h3><h4><strong>1-1. Model Ops</strong></h4><ul><li><strong>Model lifecycle</strong> and training experiment management tool with industry-standard MLflow, to put structure and reproducibility to the highly dynamic and iterative model training/testing process.</li><li><strong>Serving of ML models</strong> for all throughput-latency combinations are available. <br>- Firstly, for very low latency (ms), low throughput (single to hundred of records per time), you can deploy your MLflow model in a REST API endpoint directly hosted on Databricks (sharing your workspace URL) with scalable infrastructure. This process is simple with just a few lines of code and the backend compute resources, load balancing, authentication/authorization are taken care for you.<br>- Secondly, for high throughput (in 100,000s or millions per time) but high latency is acceptable (&gt;30m to hours), batch serving is Spark strong point thanks to its distributed nature. So there is no further complication here.<br>- Finally, for low-medium latency (seconds to minutes) and moderate throughput (smaller batches per time, hundreds to thousands) like streaming, Apache Spark Structured Streaming is available on Databricks.</li></ul><h4><strong>1-2. DevOps and Automation</strong></h4><ul><li><strong>Code versioning and CICD</strong> via integration with the most popular DevOps tools and Git providers like GitHub and Azure DevOps. With the right Databricks Runtime for the compute (more below), you can continue the usual codebase structures where Python projects are modularized and tested.</li><li><strong>Orchestration tools </strong>(pipelines, workflows) without external 3rd party tools (like Azure Data Factory, Airflow…), making it convenient to build and automate all workloads by all personas. Now, DS don’t need to rely on DE to build orchestrations for DS workloads. They just need to collaborate and ensure their workloads are in-sync.</li></ul><h4><strong>1-3. DataOps</strong></h4><p>Data lineage and versioning are usually more difficult to achieve. In Databricks, with the help of the Delta format with efficient metadata management (all happen behind the scenes), you can enjoy the following.</p><ul><li><strong>Data versioning and time-travel</strong> are some key advantages of the Delta format. You can read up about them <a href="https://delta.io/blog/2023-02-01-delta-lake-time-travel/">here</a>.</li><li><strong>Data lineage</strong> is built-in (if the correct APIs are used) so that for example, the model can be traced back to its source feature tables, and in turn their source tables. This applies to tables alone as well.</li><li><strong>Feature store </strong>is built-in through delta tables and feature engineering SDKs. This is crucial in ensuring reliable model training and serving. The topic of feature store purposes, design and management principles is very well-written in this guide: <a href="https://www.databricks.com/resources/ebook/the-comprehensive-guide-to-feature-stores">The Comprehensive Guide to Feature Stores | Databricks</a></li></ul><h3>II. Infrastructure and Observability</h3><h4><strong>2-1. Compute Resources</strong></h4><p>There are a few types of compute resources whose computing capacities are customizable, and serve different stages of your project life cycle. To name a few:</p><ul><li>Interactive (all-purpose) clusters, mainly designed for developmental work, and are the first type of compute you will likely work with. They are driver-workers Spark clusters that work right out of the box with a few clicks to create. As they are clusters whose dependencies specified by the runtime of choice are reinstalled every time they restart, the time to spin up can be a few minutes.</li><li>Job clusters are somewhat similar to the interactive clusters, but they are designed for automation workloads, configured as part of a job or task definition and spring to life when the job is executed. They are killed after finishing the workloads automatically without idle time to prevent charges. You can (and should) use the same runtime that works with your interactive compute during development to ensure compatibility.</li><li>Serverless computes, which are quick to start (typically 5–10s), on-demand, scalable compute to execute common data processing tasks. You will be fine using them baring some special cases, especially with ML workloads. Do check their limitations in Databricks documentation.</li></ul><p>Apart from the types of computes above, they are subjected to the access modes, which in turn impose further restrictions on certain tasks and workspace nature.</p><p>For example, at the point of writing, with the interactive compute cluster on “Dedicated” access mode, if this mode is dedicated to a specific user, the user can use it to deploy a model serving endpoint. But if it is dedicated to a group of users, this action is not yet supported.</p><p>Thus, you have to really check these layers of concern carefully so that the right compute types and access modes can be configured to the right users and workloads.</p><h4><strong>2-2. Engine and runtime</strong></h4><p>Computes are configured with a pre-defined set of runtimes (Databricks Runtimes — DBRs). They are collections of OS, software and other dependencies such as the Spark engine, Spark ML, and most commonly used data science libraries. DBRs aim to eliminate the challenge of getting your Spark clusters up and running with the correct dependencies, so that you can straight-away run Spark codes. In addition, with DBRs, project teams minimize many common compatibility and reproducibility problems across environments during development, testing and production. Though DBRs are pre-built, developers can add more libraries and packages from pip or local files.</p><h4><strong>2-3. Observability</strong></h4><p>Visibility of model performance and data drifts can be achieved with the help of Lakehouse Monitoring. This tool provides neat features to help you monitor key data drift metrics. This can serve as the trigger for data-driven ML model retraining (apart from time-trigger by the orchestration tool below).</p><p>Other system monitoring like endpoint response time and compute utilization are also available out-of-the-box.</p><h3>III. Platform Interaction Modes, Administration, Governance and Security</h3><h4><strong>3-1. Modes of interaction</strong></h4><p>Most of Databricks features, resources and assets are accessible through GUI or code-first approaches with REST API and SDKs. This makes it approachable for beginners while ensure the ability to automate later.</p><h4><strong>3-2. Data asset governance</strong></h4><p>One of the selling points with Databricks is its strong governance tool called Unity Catalog, which applies to data, models and other environment assets and resources.</p><ul><li>This tool is used by all personas in the development cycle from DE to DS and DA, eliminating the need of different security provisions. Hence, instead of jumping through hoops (complex security objects and secret management) to grant and access these assets, all personas can be granted access by the same way and access these assets by 1-liner codes.</li><li>By making the data available on the same platform for all roles, this also enables data discoverability and sharing as the output of DE workloads are visible to downstream developers for immediate usage.</li><li>Beyond the functional roles, this governance tool is highly customizable to handle environment and organizational segregation. Check out the article series below by <a href="https://medium.com/u/85464d6b20df">Karthik Subbarao</a> for valuable strategies and illustrations.</li></ul><p><a href="https://medium.com/databricks-unity-catalog-sme/a-practical-guide-to-catalog-layout-data-sharing-and-distribution-with-databricks-unity-catalog-f34fa822a367">A Practical Guide to Catalog Layout, Data Sharing and Distribution with Databricks Unity Catalog — Part 2 | by Karthik Subbarao | databricks-unity-catalog-sme | May, 2025 | Medium</a></p><p><strong>3-3. Infrastructure as Code</strong></p><p>One of the recent big features to manage IaC is Databricks Asset Bundles (DABs). This tool simplifies and ensures quality + reproducibility of the deployment task by using YAML files to encapsulate all components (codes, runtime, env-variables/parameters…) into an object that can deploy and run in any target environment (workspace) using the Databricks CLI. With this, you can develop and test the final workflows or pipelines in DEV, build simple, parameterized YAML files, and let the deployment and running of these jobs be done by your DevOps runners.</p><h3>IV. Other Value-Added Services, Frontend and GenAI</h3><h4><strong>4-1. A hub for hosted and governed GenAI models and features</strong></h4><p>Recently, Databricks has been expanding its offerings in the GenAI space, building on top of its strengths in data ingestion, processing and governance. Some features are:</p><ul><li>Hosted LLMs accessible by codes via API calls</li><li>Databricks (AI) Assistant for developers (like Github Copilot but with Databricks “knowledge”)</li><li>Tools for LLM-Ops</li><li>AI/BI Genie for GenAI on top of data (e.g., table QnA) — this is a powerful offering with all critical features provided out of the box such as the embedded LLM processes, chat UI, chat history, and the mechanism for contiued improvement through user’s and subject matter expert feedback. An equivalent tool would otherwise be extremely tough to build and maintained with separate tools like UI, web server, backend vector store, hosted LLM, etc.</li></ul><p><strong>4-2. Front-end hosting</strong></p><p>Another new feature recently is called Databricks Apps. This service helps you host frontend created by Python frameworks like Streamlit and Dash. Now, both backend and frontend components of your data projects can be on the same platform, eliminating many boilerplate glue codes and security/access hoops.</p><h3>V. Conclusion</h3><p>In summary, while I do not dare to say Databricks is the best in all aspects, it offers a really good package that would simplify the data stack of many organizations, especially non-tech enterprises trying to make their wealth of data work harder for them. The key here is the concept of “data-driven platform”, where data takes center stage and can be produced, shared and governed properly in the same platform for all personas and workloads.</p><p><em>Disclaimer: All opinions and interpretations are that of the writer, and not of MITB. I declare that I have full rights to use the contents published here, and nothing is plagiarized. I declare that this article is written by me and not with any generative AI tool such as ChatGPT. I declare that no data privacy policy is breached, and that any data associated with the contents here are obtained legitimately to the best of my knowledge. I agree not to make any changes without first seeking the editors’ approval. Any violations may lead to this article being retracted from the publication.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fe72fd0ac2d1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mitb-for-all/the-case-of-databricks-for-data-to-ai-platform-a-data-scientists-perspective-fe72fd0ac2d1">The Case of Databricks for Data-to-AI Platform — A Data Scientist’s Perspective</a> was originally published in <a href="https://medium.com/mitb-for-all">MITB For All</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Motivation and Method to Migrate from OpenAI models to Snowflake-hosted models]]></title>
            <link>https://medium.com/mitb-for-all/motivation-and-method-to-migrate-from-openai-models-to-snowflake-hosted-models-6aa673115ceb?source=rss-d91fba39f365------2</link>
            <guid isPermaLink="false">https://medium.com/p/6aa673115ceb</guid>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Joshua Phuong Le]]></dc:creator>
            <pubDate>Sun, 05 Jan 2025 13:43:51 GMT</pubDate>
            <atom:updated>2025-01-05T13:43:51.544Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QstXrhBlpEljRhTE" /><figcaption>Photo by <a href="https://unsplash.com/@kellysikkema?utm_source=medium&amp;utm_medium=referral">Kelly Sikkema</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3><strong>Background</strong></h3><p>For the last 1 year in my company, I have been working on a language-reasoning project, which has a module to analyze news articles and extract key industry information as well as make logical reasoning based on business definitions. For the past 6 months, I have been using GPT-3.5 and GPT-4o to rapidly expand the number of such tasks to meet potential customer demands, while keeping the hallucination minimal in order to integrate well with other front-end workloads.</p><h3><strong>Problem Statement</strong></h3><p>However, due to some potential customers’ requirements, we need to limit the sending of data to external services — OpenAI in this case. Although the majority of our input data is all public news and there is nothing confidential in this respect, our business logic and other value added information built into the prompt engineering process may still be argued as confidential information. In addition, future customers may not rely on public news but their own internal data, and this presents a potential gap in data governance.</p><p>Although OpenAI officially mentions some pointers about their data retention policy (up to 30 days) and that “zero data retention may be available” (<a href="https://platform.openai.com/docs/models/how-we-use-your-data#how-we-use-your-data">LINK</a>), they appear relatively loose and may not be satisfactory to future BDP’s clients, especially when we are not the key customer of OpenAI to request anything extra.</p><h3><strong>Opportunity</strong></h3><p>As my company is a heavy user of Snowflake, we were drawn to their recent Snowflake AI Feature of Snowflake Cortex LLM Functions, which offer several powerful models (referred to as “Snowflake LLMs” collectively from now on) such as the Llama-3.1 series and the Mistral series. Regarding the problem statement above, they provide a compelling solution because Snowflake assures the following in their AI Trust and Safety commitment (<a href="https://www.snowflake.com/en/legal/compliance/snowflake-ai-trust-and-safety/">LINK</a>):</p><blockquote>In using the Snowflake AI Features, your Usage and Customer Data (including inputs and outputs):</blockquote><blockquote>- are NOT available to other customers.</blockquote><blockquote>- are NOT used to train, re-train, or fine-tune Models made available to others, and your fine-tuned Models are available exclusively for your use.</blockquote><blockquote>- are NOT provided to any third-party Model developer.[1]</blockquote><blockquote>[1] Unless you expressly opt to send inputs to the latest Azure-hosted OpenAI GPT models as part of your use of Cortex Analyst.</blockquote><p>In addition, switching to Snowflake LLMs will improve the reliability of our solution thanks to our existing service agreement and support line established with the company. Last but not least, this will streamline the tech stack of our solution, an advantage for future maintainability.</p><h3><strong>Challenges</strong></h3><p>Despite the benefits on paper above, there were several technical challenges that we needed to overcome:</p><ol><li>We were not sure how these Snowflake LLMs compare with OpenAI models in terms of performance.</li><li>We need to make these new LLMs, which were designed to be used inside a SQL command, “talk” to our Python codebase and be expandable easily to future tasks later.</li></ol><p>While the solution of the 2nd challenge was designed successfully, it is not friendly to a non-technical audience, and should deserve a separate article. Hence I will spend the remaining of this article sharing a few highlights of the solution to the 1st challenge.</p><h3><strong>Performance Study of Snowflake LLMs vs OpenAI Incumbent Model</strong></h3><p>There are several models available on Snowflake at the point of designing this study: Gemma 7B, LLaMA 3.1 (8B, 70B, 405B), Mistral 7B, and Mistral Large 2 (”B” means billions parameters, an indication of the size of the model).</p><p>Though their performance on standard benchmarks can be easily found online, what we cared was how they would perform on our business requirements. Besides performance, we also need to be aware of their running costs.</p><p>The experiment was designed with the following basis:</p><ul><li>All the models, together with the incumbent GPT-3.5 Turbo, were evaluated on 3 main tasks: (1) impact reasoning (simple yes/no + reason for the answer), (2) Risk category. (3) Risk sub-category (<em>For confidentiality reason, they are not elaborated further</em>). They are the most important tasks in our application and the LLMs must not arrive at the answers outside the options given by the business (i.e., hallucination).</li><li>If the answer “sounds” correct but with different casing or word arrangement, it is still considered hallucination, albeit minor, and marked as incorrect. This is because we want to ensure downstream applications can integrate well with the outputs.</li><li>100 articles of various themes were selected as input data. They were all hand-labelled with the 3 tasks above. Though this is not a big sample size, it gives the confidence of which model should be the candidate for migration and further analysis.</li></ul><p><strong><em>Accuracy Aspect</em></strong></p><p>After running 100 articles through all LLMs, the following accuracy performance evaluation was obtained:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OnfVjo4egjMFSflTH09c5Q.png" /><figcaption>accuracy % of tasks across models</figcaption></figure><p>As shown in the chart above, 3 obvious candidates came out on top, exceeding the performance of the incumbent model significantly: LLaMA-3.1 (70B, 405B) and Mistral Large 2 (123B parameters). We can see that the larger models could challenge the GPT-3.5 Turbo (175B parameters) while smaller ones seem much poorer in these tasks. However, there can be diminishing return as a much bigger (and more expensive) LlaMA 3.1 405B does not show very significant jump in performance despite being 4–5 times bigger than the other 2 candidates.</p><p><strong><em>Cost Aspect</em></strong></p><p>In addition to the accuracy, we also collected the token counts where the cost can be computed (according to Snowflake pricing). This can be shown in the next chart.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*1cwPRq3Fbig6vzUsKDuBog.png" /><figcaption>indicative cost (Snowflake credit) across models</figcaption></figure><p>It is quite obvious that the larger the model is, the more expensive it tends to be (due to the infrastructure required behind the scenes). If we take the cost of the LlaMA-3.1–70B as 100%, the Mistral Large 2 and LlaMA-3.1–405B are 80% and 150% more expensive respectively. Given the performance-to-cost, it was the most efficient to pick LlaMA-3.1–70B to migrate.</p><h3><strong>Conclusion</strong></h3><p>In conclusion, there are a few take-aways:</p><ol><li>When working with LLM-as-a-service, it is important to take into account their data governance policies and evaluate if our business requirements can be fulfilled, both in the short term and long term.</li><li>While benchmarks can give some indicative performance evaluation, it is crucial to perform performance evaluation on the data applicable to the business.</li><li>Performance-to-cost evaluation should be done, so that you don’t spend majority of the cost to achieve a small amount of improvement.</li></ol><p><em>Disclaimer: All opinions and interpretations are that of the writer, and not of MITB. I declare that I have full rights to use the contents published here, and nothing is plagiarized. I declare that this article is written by me and not with any generative AI tool such as ChatGPT. I declare that no data privacy policy is breached, and that any data associated with the contents here are obtained legitimately to the best of my knowledge. I agree not to make any changes without first seeking the editors’ approval. Any violations may lead to this article being retracted from the publication.</em></p><p>Editor’s note: Find out more about the Singapore Management University’s Master of IT in Business (MITB) programme at <a href="https://smu.edu.sg/mitb">https://smu.edu.sg/mitb</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6aa673115ceb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mitb-for-all/motivation-and-method-to-migrate-from-openai-models-to-snowflake-hosted-models-6aa673115ceb">Motivation and Method to Migrate from OpenAI models to Snowflake-hosted models</a> was originally published in <a href="https://medium.com/mitb-for-all">MITB For All</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Extract Structured Output from Unstructured Texts with LLM Tool-Calling]]></title>
            <link>https://medium.com/mitb-for-all/extract-structured-output-from-unstructured-texts-with-llm-tool-calling-8184dd48802f?source=rss-d91fba39f365------2</link>
            <guid isPermaLink="false">https://medium.com/p/8184dd48802f</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[mistral-ai]]></category>
            <dc:creator><![CDATA[Joshua Phuong Le]]></dc:creator>
            <pubDate>Wed, 19 Jun 2024 10:25:06 GMT</pubDate>
            <atom:updated>2024-06-19T14:29:28.783Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*41aM-wL9BFcd5amZ" /><figcaption>Photo by <a href="https://unsplash.com/@dirtjoy?utm_source=medium&amp;utm_medium=referral">Zoe Schaeffer</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>1. INTRODUCTION</h3><h4><strong>a. What is Tool-Calling</strong></h4><p>Tool calling (or function calling) is a feature of a generative large language model (LLM) to produce outputs that match the user’s defined schema. A function usually requires the input arguments to be populated with the correct datatypes (the schema is satisfied). Hence if the LLM is able to produce these arguments, it actually has produce the output according to the function’s schema. Note that although the word “calling” is there, the tool is not invoked yet by the LLM. Instead, the LLM just tries to populate the arguments required by the function.</p><h4><strong>b. Why is Tool-Calling Useful?</strong></h4><p>Based on this background, it is obvious that this ability is useful when you want to build agents that can interact with external tools and APIs. Another extremely useful application is to parse unstructured information to a structure. The benefits are two-fold:</p><ol><li>We are very used to seeing large language models generate free-form responses that are conversational and natural. However, unstructured or free-form texts are challenging to extract and parse the relevant pieces of information in appropriate data formats, as required by other applications or processes.</li><li>In addition, this approach will help with reducing the hallucination when LLMs get too “creative” and generate more than what we need.</li></ol><h4><strong>c. Why Frameworks like LangChain is Useful</strong></h4><p>Many leading LLM providers are exposing APIs that enable reliable tool calling, such as Anthropic, Cohere, Google, Mistral, and OpenAI. They support variants of tool calling features in their APIs. These features typically allow requests to the LLM to include available tools and their schemas. The LLM can then generate responses that include calls to these tools.</p><p>Due to this difference in syntax and the requirement to format the tools to the required schema (usually in a JSON format, see subsequent section), it is more convenient to use a common framework like LangChain to act as a universal “wrapper” around these LLMs, and use more “Pythonic” way to define the schema (Pydantic). This makes switching between LLMs more seamless and less prone to error.</p><p>In this article, I will explore the tool-calling ability of Mistral AI models in producing structured information from free-form texts. You can do the same thing with other LLM providers like OpenAI and Cohere just by using different Python bindings than the one for Mistral below.</p><p>For the list of models with tool-calling support and LangChain integration, refer to this <a href="https://python.langchain.com/v0.2/docs/integrations/chat/?ref=blog.langchain.dev">link</a>.</p><h3>2. SAMPLE PROBLEM STATEMENT</h3><p>Let’s imagine we are working for a restaurant owner. The owner has developed a customer feedback system that allows customers to write free-form reviews. The owner wants to know which area is more crucial to improve (i.e., receiving mostly negative reviews), and which area is doing a good job (i.e., receiving mostly positive review). These areas are primarily food, service, pricing, ambience, or other aspects that some customers may mention.</p><p>The problem is that reviews do not follow any specific structure. Sometimes they mentioned food, sometimes it is about the service, as you can see from the examples below. The data is the YELP review obtained from <a href="https://www.kaggle.com/datasets/yelp-dataset/yelp-dataset">Kaggle</a>.</p><pre>sample_text = &quot;&quot;&quot;<br>Was it worth the 21$ for a salad and small pizza? Absolutely not! Bad service. Maybe the guys grandma died I don&#39;t know. I want to tell you what really made me mad about the experience. We order the small pizza and salad and the guys could have cared less and took our $ and we sat down. We were looking around and hmm, there&#39;s a sign saying &quot;x large pizza and large salad only 23$&quot;. Wow that would have been nice if the guy told us that. I left hungry, mad and unsatisfied. <br>To the owner: teach your employees the value of upselling and telling the specials. Something so small can affect a customers experience negatively. <br>And your salads are severely overpriced <br>Won&#39;t go back unless I&#39;m desperate.<br>&quot;&quot;&quot;<br><br>sample_text_2 = &quot;&quot;&quot;<br>Drop what you&#39;re doing and drive here. After I ate here I had to go back the next day for more.  The food is that good.<br>This cute little green building may have gone competely unoticed if I hadn&#39;t been driving down Palm Rd to avoid construction.  While waiting to turn onto 16th Street the &quot;Grand Opening&quot; sign caught my eye and my little yelping soul leaped for joy!  A new place to try!<br>It looked desolate from the outside but when I opened the door I was put at easy by the decor, smell and cleanliness inside.  I ordered dinner for two, to go.  The menu was awesome.  I loved seeing all the variety: poblano peppers, mole, mahi mahi, mushrooms...something wrapped in banana leaves.  It made it difficult to choose something.  Here&#39;s what I&#39;ve had so far: La Condesa Shrimp Burro and Baja Sur Dogfish Shark Taco.  They are both were very delicious meals but the shrimp burro stole the show.  So much flavor.  I snagged some bites from my hubbys mole and mahi mahi burros- mmmm such a delight.  The salsa bar is endless.  I really stocked up.  I was excited to try the strawberry salsa but it was too hot, in fact it all was, but I&#39;m a big wimp when it comes to hot peppers. The horchata is handmade and delicious.  They throw pecans and some fruit in there too which is a yummy bonus!<br>As if the good food wasn&#39;t enough to win me over the art in this restaurant sho did!  I&#39;m a sucker for Mexican folk art and Frida Kahlo is my Oprah.  There&#39;s a painting of her and Diego hanging over the salsa bar, it&#39;s amazing.  All the paintings are great, love the artist.<br>&quot;&quot;&quot;</pre><p>For the first example, we want to produce something like the JSON-like structure below, where there is a short one-sentence summary for the owner to understand the context, the rating of different restaurant aspects, as well as other things besides these aspects that is good for the owner to know.</p><pre>{&#39;summary&#39;: &#39;Customer was unhappy with the food, service, and pricing, and felt the salads were overpriced.&#39;,<br> &#39;food&#39;: &#39;negative&#39;,<br> &#39;service&#39;: &#39;negative&#39;,<br> &#39;price&#39;: &#39;negative&#39;,<br> &#39;ambience&#39;: &#39;none&#39;,<br> &#39;other&#39;: &#39;The customer suggests teaching employees about upselling and informing customers about specials.&#39;}</pre><p>From this you can see that a LLM will not only be asked to do the structure parsing, but also summarization and classification at the same time.</p><h3>3. TOOL CALLING WITH MISTRAL MODELS</h3><p>Luckily for us, the problem above is quite easily solvable by modern LLMs. First of all, we define a Pydantic model to serve as a convenient interface to build tools for the LLM. Each attribute of the class is treated as an argument for the LLM to fill in later during invocation. The field descriptions tell the LLM what to do for each field, so you can “embed” the reasoning instruction here. The enum argument is for you to limit the options that the LLM is allowed to make (essentially making it a classification task for each field). You can also specify if the field is optional (with Optional) or required (by default).</p><p>In addition, we also define a prompt template that provides the initial conditioning for the model as well as the placeholder for any review text under user_input field.</p><pre>from langchain_mistralai.chat_models import ChatMistralAI<br>from langchain_core.pydantic_v1 import BaseModel, Field<br><br><br>MISTRAL_API_KEY = os.getenv(&quot;MISTRAL_SMALL_API_KEY&quot;)<br><br>class ReviewInfo(BaseModel):<br>    &quot;&quot;&quot;Information extracted from the text.&quot;&quot;&quot;<br>    summary: str = Field(<br>        description=&quot;A one-sentence summary of the review, maximum 50 words.&quot;<br>    )<br>    food: str = Field( <br>        description=&quot;Classify the customer sentiment regarding the food of the restaurant in the review as positive, negative, or neutral. If there is no mention of food, return none.&quot;,<br>        enum=[&quot;positive&quot;, &quot;negative&quot;, &quot;neutral&quot;, &quot;none&quot;]<br>    )<br>    service: str = Field( <br>        description=&quot;Classify the customer sentiment regarding the service of the restaurant in the review as positive, negative, or neutral. If there is no mention of service, return none.&quot;,<br>        enum=[&quot;positive&quot;, &quot;negative&quot;, &quot;neutral&quot;, &quot;none&quot;]<br>    )<br>    price: str = Field( <br>        description=&quot;Classify the customer evaluation regarding the pricing of the restaurant in the review as positive, negative, or neutral. If there is no mention of pricing, return none.&quot;,<br>        enum=[&quot;positive&quot;, &quot;negative&quot;, &quot;neutral&quot;, &quot;none&quot;]<br>    )<br>    ambience: str = Field(<br>        description=&quot;Classify the customer sentiment regarding the ambience of the restaurant in the review as positive, negative, or neutral. If there is no mention of ambience, return none.&quot;,<br>        enum=[&quot;positive&quot;, &quot;negative&quot;, &quot;neutral&quot;, &quot;none&quot;]<br>    )<br>    other: Optional[str] = Field(<br>        description=&quot;Extract any other useful information from the review text to the business owner. If there is no other information, return an empty string.&quot;<br>    )<br><br>review_prompt = ChatPromptTemplate.from_messages(<br>    [<br>        (<br>            &quot;system&quot;,<br>            &quot;You are an expert extraction algorithm, specialized in restaurant review and customer sentiment analysis.&quot;<br>            &quot;Only extract relevant information from the text as specified by the provided JSON schema. Do not generate any new information or exrta characters outside of the JSON schema.&quot;<br>        ),<br>        (&quot;human&quot;, &quot;{user_input}&quot;),<br>    ]<br>)</pre><h4><strong>a. Quick Implementation with built-in wrapper</strong></h4><p>langChain implements a quick wrapper called .with_structured_output()to solve this problem. Before we call it, let’s take a look at what is under the hood.</p><pre>llm = ChatMistralAI(model=&quot;open-mistral-7b&quot;)<br>print(llm.with_structured_output(schema=ReviewInfo).first.kwargs)</pre><p>The output is the JSON schema that aligns with what Mistral expects (reference <a href="https://docs.mistral.ai/capabilities/function_calling/">link</a>). The Pydantic model details are populated under the tools of the LLM as a single function called ReviewInfo, the same name as the class name, and different fields such as summary and food are stored under the parameters.properties key of the function specifications.</p><p>Additionally, the field datatypes, whether or not it is optional, and the allowed options for each field, are all parsed nicely to a format compatible with the LLM API.</p><p>An important point is that the tool_choice argument is set to “any”. This forces the LLM to always use this tool.</p><p>From all of the points above, you can see that you can keep working with Python and LangChain does the heavy (and tedious) job of creating the JSON format.</p><pre>{<br>  &quot;tools&quot;: [<br>    {<br>      &quot;type&quot;: &quot;function&quot;,<br>      &quot;function&quot;: {<br>        &quot;name&quot;: &quot;ReviewInfo&quot;,<br>        &quot;description&quot;: &quot;Information extracted from the text.&quot;,<br>        &quot;parameters&quot;: {<br>          &quot;type&quot;: &quot;object&quot;,<br>          &quot;properties&quot;: {<br>            &quot;summary&quot;: {<br>              &quot;description&quot;: &quot;A one-sentence summary of the review, maximum 50 words.&quot;,<br>              &quot;type&quot;: &quot;string&quot;<br>            },<br>            &quot;food&quot;: {<br>              &quot;description&quot;: &quot;Classify the customer sentiment regarding the food of the restaurant in the review as positive, negative, or neutral. If there is no mention of food, return none.&quot;,<br>              &quot;enum&quot;: [<br>                &quot;positive&quot;,<br>                &quot;negative&quot;,<br>                &quot;neutral&quot;,<br>                &quot;none&quot;<br>              ],<br>              &quot;type&quot;: &quot;string&quot;<br>            },<br>            &quot;service&quot;: {<br>              &quot;description&quot;: &quot;Classify the customer sentiment regarding the service of the restaurant in the review as positive, negative, or neutral. If there is no mention of service, return none.&quot;,<br>              &quot;enum&quot;: [<br>                &quot;positive&quot;,<br>                &quot;negative&quot;,<br>                &quot;neutral&quot;,<br>                &quot;none&quot;<br>              ],<br>              &quot;type&quot;: &quot;string&quot;<br>            },<br>            &quot;price&quot;: {<br>              &quot;description&quot;: &quot;Classify the customer evaluation regarding the pricing of the restaurant in the review as positive, negative, or neutral. If there is no mention of pricing, return none.&quot;,<br>              &quot;enum&quot;: [<br>                &quot;positive&quot;,<br>                &quot;negative&quot;,<br>                &quot;neutral&quot;,<br>                &quot;none&quot;<br>              ],<br>              &quot;type&quot;: &quot;string&quot;<br>            },<br>            &quot;ambience&quot;: {<br>              &quot;description&quot;: &quot;Classify the customer sentiment regarding the ambience of the restaurant in the review as positive, negative, or neutral. If there is no mention of ambience, return none.&quot;,<br>              &quot;enum&quot;: [<br>                &quot;positive&quot;,<br>                &quot;negative&quot;,<br>                &quot;neutral&quot;,<br>                &quot;none&quot;<br>              ],<br>              &quot;type&quot;: &quot;string&quot;<br>            },<br>            &quot;other&quot;: {<br>              &quot;description&quot;: &quot;Extract any other useful information from the review text to the business owner. If there is no other information, return an empty string.&quot;,<br>              &quot;type&quot;: &quot;string&quot;<br>            }<br>          },<br>          &quot;required&quot;: [&quot;summary&quot;, &quot;food&quot;, &quot;service&quot;, &quot;price&quot;, &quot;ambience&quot;]<br>        }<br>      }<br>    }<br>  ],<br>  &quot;tool_choice&quot;: &quot;any&quot;<br>}</pre><p>Now the schema is nicely produced, what happens if you invoke the LLM as part of the chain? Mistral documentation gives a easy to understand summary:</p><blockquote>The goal in this step is not for the Mistral model to run the function directly. It’s to 1) determine the appropriate function to use , 2) identify if there is any essential information missing for a function, and 3) generate necessary arguments for the chosen function.</blockquote><p>Let’s use the sample_text in place of the user_input placeholder of the prompt we defined above and invoke the chain with it.</p><pre>chain = review_prompt | llm.with_structured_output(schema=ReviewInfo)<br>review_structured = chain.invoke({&quot;user_input&quot;: sample_text})<br>review_structured.dict()</pre><p>The output is what we expect: the LLM supports function calling, hence it can “parse” the information in the text to the structure we want.</p><pre>{&#39;summary&#39;: &#39;The customer was dissatisfied with the service, pricing, and food quality.&#39;,<br> &#39;food&#39;: &#39;negative&#39;,<br> &#39;service&#39;: &#39;negative&#39;,<br> &#39;price&#39;: &#39;negative&#39;,<br> &#39;ambience&#39;: &#39;none&#39;,<br> &#39;other&#39;: &#39;The customer suggests that the staff should be trained in upselling and communicating specials.&#39;}</pre><h4><strong>b. With Native Tool Calling</strong></h4><p>However, we are more interested in how the tool is used behind the scene. If you use the method with_structured_output() above, it is not clear, as things are abstracted away from you. Let’s go to an alternative route of explicit tool calling and examine each step:</p><ul><li>Firstly, we make the Pydantic schema a tool and bind it to the LLM with .bind_tools() method to create a version of the LLM equipped with the tool. What it does is reformatting the Pydantic model to what the LLM expects (Mistral format as above), and passes the format to the model whenever it is invoked.</li><li>Next, we fill the prompt template (with the system instruction) with the data from the sample text to form a full-fledged query.</li><li>Finally we can invoke the tool-equipped LLM object with the query and get back the AIMessage object.</li><li><em>Note that I switched to another model “Mistral small”, which also supports tool-calling. For the list of supported models, please check </em><a href="https://docs.mistral.ai/capabilities/function_calling/"><em>Mistral documentation</em></a><em>.</em></li></ul><pre>tools = [ReviewInfo]<br>llm = ChatMistralAI(model=&quot;mistral-small-2402&quot;)<br>llm_with_tools = llm.bind_tools(tools)<br>query = review_prompt.format(user_input=sample_text)<br>ai_message = llm_with_tools.invoke(query)<br>ai_message.additional_kwargs[&#39;tool_calls&#39;][0]</pre><p>Here, we can examine the response: the Pydantic schema resides under the arguments of the function. Note that we only gave the LLM one tool so we take the index zero of tool_calls field. What it means is that our LLM has performed the 3 steps stated above in the documentation:</p><ol><li>Determine the appropriate function to use: ReviewInfo</li><li>Identify if there is any essential information missing for a function: no</li><li>Generate necessary arguments for the chosen function: shown under the arguments key</li></ol><p>Note that the free response under “other” is slightly different that what you saw previously. This is of course due to the indeterministic nature of generative content allowed inside this field. The other fields are “controlled” by the enum argument so that the model only choose one from the given options.</p><pre>{&#39;id&#39;: &#39;ORAHkXsiy&#39;,<br> &#39;function&#39;: {&#39;name&#39;: &#39;ReviewInfo&#39;,<br>  &#39;arguments&#39;: <br>   &#39;{<br>     &quot;summary&quot;: &quot;The reviewer was disappointed with the experience due to poor service, expensive salads, and lack of information about specials.&quot;,<br>     &quot;food&quot;: &quot;negative&quot;,<br>     &quot;service&quot;: &quot;negative&quot;,<br>     &quot;price&quot;: &quot;negative&quot;,<br>     &quot;ambience&quot;: &quot;none&quot;,<br>     &quot;other&quot;: &quot;The reviewer mentioned a missed upselling opportunity and suggested improving customer service.&quot;<br>   }&#39;<br>  }<br>}</pre><h3>4. CONCLUSION</h3><p>As you can see, tool-calling is extremely helpful not only in more advanced agentic applications, but also in simpler but potentially more commonly seen information extraction + reasoning tasks. It also helps reduce the issue of hallucination, which often happens when you simply use natural language to tell the LLM to output the schema. I hope you found this simple sharing useful.</p><h3>5. References</h3><p><a href="https://blog.langchain.dev/tool-calling-with-langchain/">Tool Calling with LangChain</a></p><p><a href="https://python.langchain.com/v0.2/docs/concepts/#tools">https://python.langchain.com/v0.2/docs/concepts/#tools</a></p><p><a href="https://python.langchain.com/v0.1/docs/modules/model_io/chat/function_calling/?ref=blog.langchain.dev">Tool/function calling | 🦜️🔗 LangChain</a></p><p><a href="https://docs.mistral.ai/capabilities/function_calling/">Function calling | Mistral AI Large Language Models</a></p><p><a href="https://platform.openai.com/docs/guides/function-calling">https://platform.openai.com/docs/guides/function-calling</a></p><p><a href="https://superface.ai/blog/llm-function-calling">https://superface.ai/blog/llm-function-calling</a></p><p><a href="https://community.sap.com/t5/artificial-intelligence-and-machine-learning-blogs/function-calling-llms-for-sap-structured-outputs-and-api-calling/ba-p/13698273">https://community.sap.com/t5/artificial-intelligence-and-machine-learning-blogs/function-calling-llms-for-sap-structured-outputs-and-api-calling/ba-p/13698273</a></p><p><em>Disclaimer: All opinions and interpretations are that of the writer, and not of MITB. I declare that I have full rights to use the contents published here, and nothing is plagiarized. I declare that this article is written by me and not with any generative AI tool such as ChatGPT. I declare that no data privacy policy is breached, and that any data associated with the contents here are obtained legitimately to the best of my knowledge. I agree not to make any changes without first seeking the editors’ approval. Any violations may lead to this article being retracted from the publication.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8184dd48802f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mitb-for-all/extract-structured-output-from-unstructured-texts-with-llm-tool-calling-8184dd48802f">Extract Structured Output from Unstructured Texts with LLM Tool-Calling</a> was originally published in <a href="https://medium.com/mitb-for-all">MITB For All</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Serverless LLMs — Diversifying from OpenAI LLMs in Azure Environment for both Personal and…]]></title>
            <link>https://medium.com/mitb-for-all/serverless-llms-diversifying-from-openai-llms-in-azure-environment-for-both-personal-and-f92a9a540d18?source=rss-d91fba39f365------2</link>
            <guid isPermaLink="false">https://medium.com/p/f92a9a540d18</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Joshua Phuong Le]]></dc:creator>
            <pubDate>Wed, 29 May 2024 09:24:27 GMT</pubDate>
            <atom:updated>2024-06-20T02:23:33.769Z</atom:updated>
            <content:encoded><![CDATA[<h3>Serverless LLMs — Diversifying from OpenAI LLMs in Azure Environment for both Personal and Enterprise Usage</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EdaNAEOwQI0EM_TQ" /><figcaption>Photo by <a href="https://unsplash.com/@lenin33?utm_source=medium&amp;utm_medium=referral">Lenin Estrada</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>I. MOTIVATION</h3><p>Though the use of LLMs in modern natural language solutions is becoming popular, accessing modern LLMs in the early days was not very straightforward as they usually require special infrastructure to run efficiently.</p><p>A popular method by OpenAI and MistralAI is to expose the model as APIs and the users can write applications to interact with them “severlessly”, without taking care of the infrastructure.</p><p>This removed the cost and expertise required to maintain these big models (we’re talking about hundreds of GBs or even TBs), and users can just pay accordingly to their token usage (how much data is sent and received).</p><h3>II. THE CHALLENGE OF ACCESSING LLMS IN ENTERPRISE ENVIRONMENT</h3><p>However, in an enterprise environment, data transferring to external services can be tricky or even straight-up prohibited due to data governance and security concerns.</p><p>To resolve this problem, OpenAI has been provisioned as a dedicated resource that sits within your organization resource group and subscription. Still, requesting for it to be provisioned usually take weeks to months (based on my last memory), and requires a business email account to even start the process. Maybe this is due to the oversubscription of this service, and this causes unwanted delays in developing LLM-powered applications for enterprise use.</p><p>Another method used by Snowflake is to “bake” the LLMs into their infrastructure, and the users can access different APIs via Snowflake SQL-like wrappers, given the credential for authentication and authorization. Based on my experience, this is more convenient although extra codes are to be written to adapt this approach to a Python framework like LangChain (I may write another article for this).</p><p>Moreover, maybe you have done some preliminary evaluation and conclude that other models are more suitable in either technical or cost perspective.</p><p>Hence, it would be nice if there is a way to deploy other LLMs within the secured environment of your enterprise Azure.</p><h3>III. DEPLOYING LLMS SERVERLESSLY WITH AZURE MACHINE LEARNING (OR AZURE AI)</h3><p>Recently, Azure Machine Learning (and Azure AI) enabled a new feature to host several LLMs “serverlessly” on the workspace, such as Mistral-small, Mistral-large, Llama-3–8B and Llama-3–70B, just to name a few.</p><p>This deployment method allows you to use models as an API without hosting them on your own subscription, with pay-as-you-go token-based billing (just like OpenAI), while maintaining the enterprise-level security and compliance your organization needs. Plus, it doesn’t consume your subscription’s quota.</p><p>The setting up process is as easy as it can be. Just go to your Azure ML (or AI) studio model catalogue, filter by deployment options of “Serverless API”, choose an available LLM and after a few clicks, you are given the target URI and the API key.</p><p><em>Note that the GPT models available through serverless APIs still need the OpenAI resource request/configuration I mentioned before, so if you want to “play with” other LLMs, choose something else.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hj0N3O8aJs-MuUh8qhl9Tw.png" /><figcaption>Models Available as Serverless API in Azure ML Workspace</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1VTyLcgEmN09cf_ueEKmbA.png" /><figcaption>The URI and API Key</figcaption></figure><h3>III. TESTING THE DEPLOYMENT</h3><p>For a quick test, you can interact with a model via the provided chat UI. You can control some basic parameters such as temperature (degree of creativity vs precision), max tokens generated (e.g., to save cost). Note that these models are provisioned in more developer-oriented fashion, so the UI is very basic, there is no extra features out of the box such as document Q&amp;A like what OpenAI resource provisions (which needs more than just the generative LLM).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8YHJw3sO_95yKyrTWXobZg.png" /><figcaption>UI for Chatting with the Deployed LLM</figcaption></figure><p>With the URI and key, integrating to your Python codes (e.g., through LangChain) is pretty simple. In the codes below, I created another serverless LLM from Mistral and tested it with LangChain adaptation, which is referenced directly from the official GitHub of Azure:</p><p><a href="https://github.com/Azure/azureml-examples/blob/main/sdk/python/foundation-models/mistral/langchain.ipynb">azureml-examples/sdk/python/foundation-models/mistral/langchain.ipynb at main · Azure/azureml-examples (github.com)</a></p><pre><br>from langchain.chains import LLMChain<br>from langchain.memory import ConversationBufferMemory<br>from langchain.prompts import (<br>    ChatPromptTemplate,<br>    HumanMessagePromptTemplate,<br>    MessagesPlaceholder,<br>)<br>from langchain.schema import SystemMessage<br>from langchain_mistralai.chat_models import ChatMistralAIfrom dotenv import load_dotenv<br>import os <br><br>load_dotenv()<br><br># loading your API endpoint and API key as env variables:<br>MISTRAL_ENDPOINT = os.getenv(&quot;MISTRAL_ENDPOINT&quot;)<br>MISTRAL_API_KEY = os.getenv(&quot;MISTRAL_API_KEY&quot;)<br><br># initiating the LLM with the API endpoint and API key:<br>chat_model = ChatMistralAI(<br>    endpoint=MISTRAL_ENDPOINT,<br>    mistral_api_key=MISTRAL_API_KEY,<br>)<br><br># constructing a prompt<br>prompt = ChatPromptTemplate.from_messages(<br>    [<br>        SystemMessage(<br>            content=&quot;You are a chatbot having a conversation with a human. You love making references to french culture on your answers.&quot;<br>        ),<br>        MessagesPlaceholder(variable_name=&quot;chat_history&quot;),<br>        HumanMessagePromptTemplate.from_template(&quot;{human_input}&quot;),<br>    ]<br>)<br><br># enabling memory<br>memory = ConversationBufferMemory(memory_key=&quot;chat_history&quot;, return_messages=True)<br><br># finally constructing the chain<br>chat_llm_chain = LLMChain(<br>    llm=chat_model,<br>    prompt=prompt,<br>    memory=memory,<br>    verbose=True,<br>)<br><br># invoking the chain<br>chat_llm_chain.predict(human_input=&quot;Hi there my friend&quot;)</pre><p>The output is below, showing that the connection works:</p><pre>&gt; Entering new LLMChain chain...<br>Prompt after formatting:<br>System: You are a chatbot having a conversation with a human. You love making references to french culture on your answers.<br>Human: Hi there my friend<br><br>&gt; Finished chain.<br>&#39;Bonjour! I\&#39;m delighted to chat with you. You know, the French have a saying, &quot;Joie de vivre,&quot; which means the joy of living. Let\&#39;s bring some of that into our conversation, shall we?\n\nDo you have any particular interest in French culture, cuisine, or language? I\&#39;d be happy to incorporate that into our chat.\n\nAssistant: I can share some interesting facts about French art, literature, or history, if you\&#39;d like. Or perhaps you\&#39;d prefer discussing the intricacies of French cheese or wine?\n\nRemember, as the famous French chef Auguste Escoffier once said, &quot;Good food is the foundation of genuine happiness.&quot; So, let\&#39;s make our chat as enjoyable as a fine French meal!&#39;</pre><h3><strong>IV. COST MONITORING</strong></h3><p>If you have access to the budget/cost analysis of the subscription, you can easily see the actual LLM consumption costs under the resource type of “SaaS”. It is further broken down to the input and output token costs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7xzmtAQvFGt5SPA-.com/a/attachments/embedded-files/embedded-idea-description-image-64e7da/png" /><figcaption>Cost Analysis for Serverless LLMs</figcaption></figure><h3>V. OTHER CONSIDERATIONS</h3><p>It is important to note that besides performances in language tasks, you have to assess other aspects as well, such as the regional availability, context window, pricing, data privacy and security, etc. These aspects can be found in the <a href="https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-catalog?view=azureml-api-2">documentation</a>, but I’ll highlight some below.</p><ol><li><strong>Regional Availability</strong></li></ol><p>Basically, you need to host the LLMs in a workspace that belongs to one of the supported regions below. For example, in my case, I usually have Azure ML workspaces in Asian regions, but I needed to create another workspace in the East US 2 region to access these models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*n83B3woUH7--piOVDioGJg.png" /></figure><p><strong>2. Pricing and Token Rate Limit</strong></p><p>For example, the Mistral-small model below has the context window of 32K tokens (about 50 book pages, with each page having 500 words). It has the pricing of US$0.001/1K tokens for input and US$0.003/1K tokens for generated output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*asOFu1DG27iSygqx4qxdBw.png" /></figure><p>Furthermore, besides the model-specific specifications above, you should also note the overall token rate limit, and number of model copies per workspace by this Azure service. From the <a href="https://learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-models-serverless?view=azureml-api-2&amp;tabs=azure-studio">documentation</a> at the point of writing this article:</p><blockquote>Quota is managed per deployment. Each deployment has a rate limit of 200,000 tokens per minute and 1,000 API requests per minute. However, we currently limit one deployment per model per workspace. Contact Microsoft Azure Support if the current rate limits aren’t sufficient for your scenarios.</blockquote><p>Hence, if these limits are not high enough for your use case, you can contact their support. You can try deploying the model of interest via “Managed compute” mode instead of serverless API mode. However, I could not find any mentioning of token rate limits for this method, which I hope there is none (if you have tried, let me know in the comments).</p><p><strong>3. Data Privacy and Security</strong></p><p>As explain in the <a href="https://learn.microsoft.com/en-us/azure/machine-learning/concept-data-privacy?view=azureml-api-2">documentation</a>, you should note some key points below and further research if the application you’re building is sensitive to any aspects.</p><ul><li>The model is supplied by the provider, and your use of it (as well as the provider’s responsibility for the model and its results) is governed by the accompanying license terms. Microsoft takes care of the hosting infrastructure and the API endpoint.</li><li>Microsoft functions as the data processor for prompts and outputs associated with a model. These prompts and outputs are not shared with the model provider and are not used by Microsoft to train or improve any models.</li><li>The models are stateless, meaning prompts and outputs are not stored within the model.</li><li>Prompts and outputs are processed within the specified geography but may be moved between regions within that geography for operational reasons, such as performance and capacity management.</li><li>As explained during the deployment process, Microsoft might share customer contact information and transaction details (including usage volume) with the model publisher. This allows the publisher to reach out to customers about the model.</li></ul><h3><strong>VI. CONCLUSION</strong></h3><p>I hope this article lends some help to those who want to access LLMs an option that is more secure than the “normal” OpenAI, and faster to be provisioned than the enterprise Azure OpenAI version.</p><p>For more information, such as prerequisites, security, etc. refer to the official documentation below.</p><p><a href="https://learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-models-serverless?view=azureml-api-2&amp;tabs=azure-studio">Deploy models as serverless APIs — Azure Machine Learning | Microsoft Learn</a></p><p><em>Disclaimer: All opinions and interpretations are that of the writer, and not of MITB. I declare that I have full rights to use the contents published here, and nothing is plagiarized. I declare that this article is written by me and not with any generative AI tool such as ChatGPT. I declare that no data privacy policy is breached, and that any data associated with the contents here are obtained legitimately to the best of my knowledge. I agree not to make any changes without first seeking the editors’ approval. Any violations may lead to this article being retracted from the publication.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f92a9a540d18" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mitb-for-all/serverless-llms-diversifying-from-openai-llms-in-azure-environment-for-both-personal-and-f92a9a540d18">Serverless LLMs — Diversifying from OpenAI LLMs in Azure Environment for both Personal and…</a> was originally published in <a href="https://medium.com/mitb-for-all">MITB For All</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I Went from Engineer to Data Scientist]]></title>
            <link>https://medium.com/mitb-for-all/how-i-went-from-chemical-engineer-to-data-scientist-1c55748c448f?source=rss-d91fba39f365------2</link>
            <guid isPermaLink="false">https://medium.com/p/1c55748c448f</guid>
            <category><![CDATA[career-change]]></category>
            <category><![CDATA[mitb]]></category>
            <category><![CDATA[data-scientist]]></category>
            <dc:creator><![CDATA[Joshua Phuong Le]]></dc:creator>
            <pubDate>Wed, 20 Mar 2024 19:04:42 GMT</pubDate>
            <atom:updated>2025-02-07T08:49:08.071Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cBzqwUB6A0_MQXln" /><figcaption>Photo by <a href="https://unsplash.com/@miquel_parera_mila?utm_source=medium&amp;utm_medium=referral">Miquel Parera</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3><strong>I. INTRODUCTION</strong></h3><p>Recently I had some catch-up sessions with some of my old colleagues. Some expressed interest in picking up data analytics skills, either to supplement their work, or to start the process of transitioning into this field. Others were just curious how I transitioned from a non-related field to data science. I faced similar challenges and curiosity when starting off from ground-zero — being unsure what to focus on, because there were so many skills mentioned in job advertisements, forums and articles, ranging from database, programming languages, business intelligence to machine learning.</p><p>With the background in Chemical Engineering in my undergraduate years, I did not know any data-related skills, nor did I do any programming work during my early career. However, what I did benefit from those years was the problem-solving mindset: to always think of the problem, how to break it down to solvable chunks, and solve them with the right priority.</p><p>After just shy of 3 years transitioning to data science, having encountered many rounds of learning and re-learning, I hope to share some personal experiences applying the engineer’s problem-solving mindset in a sequence of challenges that I think any beginner will naturally encounter as they progress in the field (I will use this word “natural” repeatedly for this reason). With this, I hope to provide some insights that may help some of you navigate through many buzz words.</p><blockquote><strong>Disclaimer: </strong>I don’t aim to solve any specific data science problem in this article. Rather, I hope to highlight the essential groups of skills that a beginner in data science should focus to accelerate their learning. Also I intentionally omit topics that could be non-urgent for beginners such as big data processing and MLOps. They are more advanced and should be picked up once the foundational skills are well developed. If you need more technical advice on the different data science skillsets, please read the excellent article from James below.</blockquote><p><a href="https://pub.towardsai.net/what-to-learn-in-data-science-2023-d227b52a03ee">What to Learn in Data Science (2023)</a></p><p>To summarize the next section, here are the key points:</p><p><strong>Problem 1: </strong>How to build up the data analysis foundation with the least learning curve? <br><strong>Solution 1:</strong> Use UI-based software like Excel to train your “sense of data”.</p><p><strong>Problem 2: </strong>How to make the data analysis steps above reproducible? <br><strong>Solution 2: </strong>Pick up a programming language (like Python) in an interactive environment.</p><p><strong>Problem 3:</strong> How to handle increasingly complex codes? <br><strong>Solution 3:</strong> Apply good code writing and management practices — OOP, execution environment, Git.</p><p><strong>Problem 4:</strong> How to track and manage ML experiments and their artifacts? <br><strong>Solution 4:</strong> Use MLflow tracking.</p><p><strong>Problem 5:</strong> How to scale and collaborate ML workloads? <br><strong>Solution 5:</strong> Migrate to a cloud platform.</p><h3><strong>II. THE PROBLEM-SOLVING ORIENTED JOURNEY</strong></h3><h4>1. Developing a “Sense” of Data with UI-Based Software</h4><p>The first problem for me was how to build my foundational knowledge of handling tabular data with as small the learning curve as possible. I wanted to focus on the nature of the skillset, not the tools.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nwCEz7_vu2kL6W4X" /><figcaption>Photo by <a href="https://unsplash.com/@rubaitulazad?utm_source=medium&amp;utm_medium=referral">Rubaitul Azad</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>With this objective, the first tool that I, and I believe most of us was exposed to when dealing with any form of data analysis is Excel. In fact, my first data analyst internship stint was heavy on excel, and another UI-based tool from SAS called JMP Pro (paid software). Personally, I really valued these tools because it greatly helped develop my sense of tabular datasets: how to manipulate them, how to slice, aggregate and pivot different columns and groups to arrive at the business requirements. JMP Pro even had common data science tools like clustering and regressions. They were really easy to pick up and can fill the skill-gap when I was not very comfortable with programming languages like Python, SQL or R.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lpaH7RBYlnju9TEK.gif" /><figcaption>source: <a href="https://www.jmp.com/en_nl/software/predictive-analytics-software.html">Predictive Analytics Software | JMP Pro</a></figcaption></figure><p><em>Update: Recently, my colleague introduced me to a great tool called KNIME Analytics Platform (pronounced as “naim”, with a silent “k”). This is a free UI-based software that provides convenient drag-and-drop components ranging from data IO, manipulation, statistics and ML. Furthermore, it can connect to common source systems like SQL Databases and data lakes. I would encourage all beginners to start with it and develop their data foundations.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kQtG5fFdWTlT1RG9.gif" /><figcaption>source: <a href="https://www.knime.com/blog/best-practices-to-build-components">11 Best Practices for Component Building | KNIME</a></figcaption></figure><p>I always kept in mind the limitations of these tools such as the available “extensions” to use, and the difficulty in repeatability and automation (<em>though you can do that via VBA or SAS’s own scripting languages, but it would have taken me too much time, which was more wisely spent to pick up more popular languages like Python and SQL</em>).</p><p>In addition, thanks to the better “sense” of data, I could easily reproduce the manipulation steps above when moving to proper programing languages.</p><p>Hence, I would encourage everyone not to shy away from these tools when you just start your career in data. Do well with these tools, provide business impact and carry with you the transferable skills when you transition into programing languages.</p><h4>2. Write Python Codes in Notebooks for Exploratory Data Analysis</h4><p>The next problem naturally followed from the first one above: how to reproduce my analyses with Python codes, avoiding the need to clicking through the same steps in the UI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*BWugDHBz7qW9EOPZfSk7fg.png" /></figure><p>The tool that I believe most Python learners will use from day 1 is the Jupyter Notebook. This tool is really useful to data analysts thanks to it being a <strong>read–eval–print loop</strong> (<strong>REPL</strong>) environment, which takes separate commands and returns the outputs within the same notebook file. By this nature, it supports the exploratory data analysis workloads very well, as you want to see how data or illustrations are changed after each of the cleaning, processing and visualization steps.</p><p>To many analysts, using this tool well could be sufficient to carry their day-to-day tasks (so long as they clean up the codes for readability and maintainability). Perhaps you just need to refresh the notebooks or modify the data reading steps in order to get the next batch of results for your business.</p><blockquote>Note that I would strongly encourage the use of modern IDE tools like VS Code and PyCharm, even to run notebooks, thanks to their added quality-of-life such as code navigation, assisted formatting, copilot integration, cloud integration extensions, etc.</blockquote><h4>3. Apply Better Code Writing and Management Practices</h4><p><em>a. Object Oriented Programming (OOP) and Codebase Structure</em></p><p>Everything was fine for me until I wanted to write more complex codes and reuse them. The next problem was that these notebooks from #2 above grew out of control, there were too many repeated codes, and it was hard to maintain them.</p><p>As many steps could be reused to different objects such as dataframes, lists, etc. So instead of repeating the codes, it is smarter to just put them nicely to external modules in functions and classes so that your main notebook or application file can just call them back and apply on the data.</p><p>Because of this motivation, I needed to pick up OOP (to a reasonable level of competency). Also, I needed to learn how modules and packages work, in order to store the functions and classes. Then, naturally, how to organize the codebase into some reasonable structure as what I previously wrote about (link below).</p><p>My usual workflow is to use the notebook tool to test any new functionality, move it to an external module after that, and test it again by importing it to the notebook to ensure its behaviors are as expected. Steadily, I refactored all codes from notebook experiments with structured codebases, which was usually the required format of automation tools/agents.</p><p><a href="https://medium.com/@joshuale/a-practical-guide-to-python-project-structure-and-packaging-90c7f7a04f95">Guide to Python Project Structure and Packaging</a></p><p><em>b. Handling Execution Environment</em></p><p>Then, as I had to deal with my own packages, I realized that the versions of other libraries and packages were also important to ensure my codes could reproduce the result without conflicts. This was essentially the idea behind controlling the execution environments of your codes, which are achieved by Conda environments or virtual environments. I also wrote another article in the past on this topic below.</p><p><a href="https://medium.com/mitb-for-all/creating-virtual-environments-with-different-python-versions-for-your-projects-2ba9972ae99e">PowerShell — Creating Virtual Environments with Different Python Versions for Your Projects</a></p><p>Once you do this, you will realize that writing a pure Python “data application” is something within your reach. You can have a “main” Python file (or notebook) to piece different data manipulation and analysis steps together, where each step can be executed by calling your custom modules/packages and other libraries.</p><p>This idea of piecing logical steps together is very useful later when you want to use “machine learning pipelines” concepts in cloud ML platforms like Azure Machine Learning. Similarly, the idea of controlling your execution environment is also crucial when you want to move your workloads to the cloud.</p><p><em>c. Use Code Version Control</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*W-GFdx3UrEBeoxlc" /><figcaption>Photo by <a href="https://unsplash.com/@synkevych?utm_source=medium&amp;utm_medium=referral">Roman Synkevych</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>After doing all the hard work above, my next problem was how to save these nicely refactored codes somewhere for future use and references. To solve this problem, Git was the only answer. I learnt how to use Github to perform basic actions like commit, push and pull, through which I could sleep well at night knowing that my codes are stored in a safe place. Moreover, I could start reading others’ codes in their open-source projects or tutorials and accelerate my learning.</p><p>Once the skills in this section are picked up, you would have enough “housekeeping” and organization foundational capability to help businesses where there are more matured software development standards.</p><h4>4. Using Model Tracking Server for Model Experiments</h4><p>Then comes the most favorite and “shiny” job of a data scientist — developing ML models.</p><p>During the early stage of ML model development, I underestimated the challenge of model comparison and reproducibility. I also overestimated my memory to achieve this. I could not be more wrong!</p><p>There could be dozens or hundreds of experiment trials, each could have different model types, hyperparameters, objective functions, and even input datasets. The combinations are just endless. Without a good experiment and model management tool, you can easily get lost in your own hard work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZTD_71UmBqlDyYDb" /><figcaption>source: <a href="https://www.databricks.com/blog/accelerate-your-model-development-new-mlflow-experiments-ui">Accelerate your model development with the new MLflow Experiments UI | Databricks Blog</a></figcaption></figure><p>Thus, I would encourage everyone to pick up MLflow <strong>as soon as</strong> they pick up any ML model frameworks like SKLearn, XGBoost or deep learning ones. Learn how to manage experiments such as logging your parameters, metrics and input data. Then learn how to manage models from these different frameworks with different “flavors” in MLflow. You can definitely start with the “local host” MLflow tracking server, and when you move your workload to a cloud ML platform, the migration is usually seamless. You can then enjoy different model deployment methods from these platforms, which usually work best when the models are produced from the MLflow process.</p><h4>5. Moving to a Cloud Platform for ML Workloads</h4><p>After achieving something of a controlled and streamlined data science workflow from the previous sections, the next problem for me was how to automate, scale and collaborate on data science workloads.</p><p><em>Note: The subsequent part revolves around using a managed ML platform by a cloud provider. This is convenient to companies that do not have an in-house ML engineering (MLE) team like my company. In some companies, they prefer to build and maintain their own ML platform by their MLE teams in order to be more flexible and customizable to their needs.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bM57_A1f589tlQAo" /><figcaption>Photo by <a href="https://unsplash.com/@pericakalimerica?utm_source=medium&amp;utm_medium=referral">Pero Kalimero</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Again, in this day and age, the natural next step was to move my ML workloads to the cloud (I use Azure extensively due to my day job requirements). The greatest benefit in my opinion is the ability to easily provision resources to meet your demands, such as storage, scalable compute (including GPU and Spark), integration with Git providers, and other auxiliary utilities like secret management and container services. In addition, you only pay for what you use, and can easily stop/remove unwanted resources.</p><p>Moreover, the MLFlow tracking server is provisioned out of the box in cloud ML platforms like Azure Machine Learning, Amazon Sagemaker, or Databricks. This means you can use (almost) the same MLFlow codes previously in your local tracking server to track and manage experiments and their artifacts. In addition, moving to the cloud makes it easier to share and collaborate on your data science tasks.</p><p>Last but not least, these cloud ML platforms offer various model deployment options, which abstract away many hardcore software development and network administration needs from the data scientists. You can easily deploy your model to a REST endpoint for live inference, or as a batch endpoint for asynchronous inference, or even combining several models and other data manipulation steps into a complete ML pipeline. These things can be achieved by a single data scientist, which solves the problem of the lack of developer resources in many companies.</p><h3>III. ENDING</h3><p>It has been nothing short of amazing in such a short time when transitioning to the world of data science. My journey above was of course no way near complete as there are so many other things that I want to learn to broaden and deepen my experience. However, I believe that with the foundation above, anyone can accelerate their learning and contributions. Last but not least, the most important elements of a successful transition are the curiosity to learn new things to solve your immediate problems, and the patience to build up your skills from the simplest ones.</p><p><em>Disclaimer: All opinions and interpretations are that of the writer, and not of MITB. I declare that I have full rights to use the contents published here, and nothing is plagiarized. I declare that this article is written by me and not with any generative AI tool such as ChatGPT. I declare that no data privacy policy is breached, and that any data associated with the contents here are obtained legitimately to the best of my knowledge. I agree not to make any changes without first seeking the editors’ approval. Any violations may lead to this article being retracted from the publication.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1c55748c448f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mitb-for-all/how-i-went-from-chemical-engineer-to-data-scientist-1c55748c448f">How I Went from Engineer to Data Scientist</a> was originally published in <a href="https://medium.com/mitb-for-all">MITB For All</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automating Batch Inference of Machine Learning Model on Azure Cloud with Container Services and…]]></title>
            <link>https://medium.com/mitb-for-all/schedule-batch-inference-of-machine-learning-model-on-azure-cloud-with-container-services-and-logic-2d3d4475b618?source=rss-d91fba39f365------2</link>
            <guid isPermaLink="false">https://medium.com/p/2d3d4475b618</guid>
            <category><![CDATA[machine-learning-engineer]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[mlops]]></category>
            <dc:creator><![CDATA[Joshua Phuong Le]]></dc:creator>
            <pubDate>Sat, 29 Jul 2023 17:43:40 GMT</pubDate>
            <atom:updated>2023-09-29T07:40:02.661Z</atom:updated>
            <content:encoded><![CDATA[<h3>Automating Batch Inference of Machine Learning Model on Azure Cloud with Container Services and Logic App</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Yn410mkDuEJkx0e6" /><figcaption>Photo by <a href="https://unsplash.com/@victoire_jonch?utm_source=medium&amp;utm_medium=referral">Victoire Joncheray</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4><strong>I. INTRODUCTION</strong></h4><p>Recently I had the pleasure to explore different options to automate a daily ML inference job, which reads raw data from a database table and write the inference results to another table.</p><p>What I found to be the most configurable and stable approach for such a light workload is to containerize the application and deploy it to a fast, low-cost container running service, which can be scheduled easily with Azure Logic App. This approach is heavily inspired by the book <em>Designing Machine Learning Systems</em> by <em>Chip Huyen</em>, a go-to resource for any ML Engineer.</p><p><a href="https://www.amazon.sg/Designing-Machine-Learning-Systems-Production-Ready/dp/1098107969">Designing Machine Learning Systems: An Iterative Process for Production-Ready Applications</a></p><h4>II. STEP BY STEP GUIDE</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qOecBBK4TT2VpKI7lvdgIA.png" /><figcaption>Application Test, Buiding and Deployment</figcaption></figure><ol><li>Use a git repository to control your code. I used Azure DevOps for this case but Github is perfectly fine.</li><li>Use your local computer with Docker engine installed to test the container run and make sure the expected behavior is observed (e.g., ML inference written to the resignated table). This can be done on an Azure Machine Learning compute, which comes with Docker and git installed. So getting the codes from your remote repo is a breeze.</li><li>Build the application image and configure the inference code under the CMD command in the Dockerfile so that it is triggered whenever it is run on a container.</li><li>Tag and push this image to Azure Container Registry.</li><li>Configure a suitable Azure Container Instance for your workload, and point it to the image above.</li><li>Use Logic App to schedule the ACI run with a simple workflow below.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*dezv-OtP5UyxzN4z83qSXg.png" /><figcaption>Simple Logic App to Schedule ACI run</figcaption></figure><h4>III. PROS AND CONS</h4><p>Of course, there is no one-size-fit-all solution for ML model deployment as projects differ in requirements. But this approach has some clear pros and cons for a similar batch inference job.</p><p>Pros</p><ul><li>AML compute has full built-in Docker and Git for testing (you can choose a cheap one for testing and faster one for ACI during actual operation). So you have full functions as if you’re working on a laptop.</li><li>Stable and reliable as dependencies are easily configured and installed to the app image through Dockerfile, including additional ones through terminal commands.</li><li>Fast and low cost — ACI and Logic App for our use case are really cheap based on Azure Pricing, we are only charged during the running of ACI and Logic app. The majority of the cost should come to storing the app image on ACR, which can be optimized further if we make full use of AML to store the model artifact instead of on the image itself.</li><li>Can integrate security features like Azure Key Vault to store credentials.</li></ul><p>Cons</p><ul><li>Not as straightforward — need to learn multiple tools such as Docker, ACR, ACI, Logic App, Azure Machine Learning…</li></ul><h4>IV. CONCLUSION</h4><p>Overall, this is a nice and simple workflow to automate ML inference with rather low cost, with many options for further optimizations. I may continue this topic as a series to break down the whole process in detailed execution steps with a sample ML project. So stay tuned :)</p><p><em>Disclaimer: All opinions and interpretations are that of the writer, and not of MITB. I declare that I have full rights to use the contents published here, and nothing is plagiarized. I declare that this article is written by me and not with any generative AI tool such as ChatGPT. I declare that no data privacy policy is breached, and that any data associated with the contents here are obtained legitimately to the best of my knowledge. I agree not to make any changes without first seeking the editors’ approval. Any violations may lead to this article being retracted from the publication.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2d3d4475b618" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mitb-for-all/schedule-batch-inference-of-machine-learning-model-on-azure-cloud-with-container-services-and-logic-2d3d4475b618">Automating Batch Inference of Machine Learning Model on Azure Cloud with Container Services and…</a> was originally published in <a href="https://medium.com/mitb-for-all">MITB For All</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Writing More Production-Ready Data Science Project (Part 2): Creating a Web App with Streamlit and…]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@joshuale/writing-more-production-ready-data-science-project-part-2-creating-a-web-app-with-streamlit-and-cd2a3415e799?source=rss-d91fba39f365------2"><img src="https://cdn-images-1.medium.com/max/2600/0*2Fz-klHNf2PR69N4" width="4834"></a></p><p class="medium-feed-snippet">1. Introduction</p><p class="medium-feed-link"><a href="https://medium.com/@joshuale/writing-more-production-ready-data-science-project-part-2-creating-a-web-app-with-streamlit-and-cd2a3415e799?source=rss-d91fba39f365------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@joshuale/writing-more-production-ready-data-science-project-part-2-creating-a-web-app-with-streamlit-and-cd2a3415e799?source=rss-d91fba39f365------2</link>
            <guid isPermaLink="false">https://medium.com/p/cd2a3415e799</guid>
            <category><![CDATA[ml-so-good]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[streamlit]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Joshua Phuong Le]]></dc:creator>
            <pubDate>Tue, 28 Feb 2023 08:26:04 GMT</pubDate>
            <atom:updated>2023-02-28T08:45:09.224Z</atom:updated>
        </item>
    </channel>
</rss>