<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sanjit Khasnobis on Medium]]></title>
        <description><![CDATA[Stories by Sanjit Khasnobis on Medium]]></description>
        <link>https://medium.com/@khasnobis.sanjit890?source=rss-5bf4882d9b72------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*gOxwJEpFDszNFJT_VrzO_g.jpeg</url>
            <title>Stories by Sanjit Khasnobis on Medium</title>
            <link>https://medium.com/@khasnobis.sanjit890?source=rss-5bf4882d9b72------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 01:52:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@khasnobis.sanjit890/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Revisit — Create Partions using py-spark in Databricks]]></title>
            <link>https://medium.com/@khasnobis.sanjit890/revisit-create-partions-using-py-spark-in-databricks-cb9112048270?source=rss-5bf4882d9b72------2</link>
            <guid isPermaLink="false">https://medium.com/p/cb9112048270</guid>
            <dc:creator><![CDATA[Sanjit Khasnobis]]></dc:creator>
            <pubDate>Mon, 21 Sep 2026 19:51:20 GMT</pubDate>
            <atom:updated>2026-09-21T19:51:20.460Z</atom:updated>
            <content:encoded><![CDATA[<p>Today we will create partitions using pyspark in Databricks.</p><p>We have created catalog -&gt; volume</p><p>catalog -&gt; sparkcatalog</p><p>volume -&gt; source</p><p>I have uploaded a file name Order.csv.</p><p>Now i will write a pyspark program in Databricks to create multiple files using partition.</p><p>First let us read the csv data as below -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pHWe3bXO0PPyTjL1XpHp8g.png" /></figure><p>Now we can see data is loaded into spark Dataframe -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yJnmQmntVouojBe2x1z-pw.png" /></figure><p>Good, Now let us create the keys for partition.</p><p>We will use Order date for partiotion logic by creating keys as-</p><p>year, month, day. The code will look like as below -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*arH6P043urUsZE-anShG7A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ev8xwqvpF6TRe1JWsZigwA.png" /><figcaption><strong>year, month, day column added using withColumn for using as partion key</strong></figcaption></figure><p>Lets now create the partion in order_partions folder under source volume by running below code -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZzBcBNOlhONLCDLbxrse_g.png" /></figure><p>Good, now we can go to folder and check if the data is created as per PartionBy logic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jkBa995Sb2lzc4gSaHXJrA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nuz_yi_Y4w-Yp0erfSimOg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RZOG2e_NUm6FScYiN-ILCw.png" /></figure><p>Databricks have created as requested the files in year/month/days partions.</p><p>It have created 27 partitions like 2025/12/01 .. 2025/12/02 …</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wl5MSh51my8zySA15mC04g.png" /></figure><p>Question is why we have created the partion.</p><p>The real reason is for partition pruning.</p><p><strong>Partition pruning</strong> in Spark is the process of reading only the relevant partitions that match a query filter, instead of scanning all partitions.</p><p>Let me write the code and prove you that as below -</p><p>you can see below code only scanned data for 25/12/2025 only 1 file scanned and it fetched 29 records as below rather than scanning all files -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*55IeFAX1aRoW-w7sAK9Zzw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RBhvVoVAFvNarrWwEw5XNQ.png" /></figure><p>Hope you have enjoyed the article!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cb9112048270" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Revisiting py-Spark Lazy evaluation using Databricks]]></title>
            <link>https://medium.com/@khasnobis.sanjit890/revisiting-py-spark-lazy-evaluation-using-databricks-649f13ca346c?source=rss-5bf4882d9b72------2</link>
            <guid isPermaLink="false">https://medium.com/p/649f13ca346c</guid>
            <dc:creator><![CDATA[Sanjit Khasnobis]]></dc:creator>
            <pubDate>Mon, 21 Sep 2026 11:53:32 GMT</pubDate>
            <atom:updated>2026-09-21T11:53:32.883Z</atom:updated>
            <content:encoded><![CDATA[<p>Today we will revise the concept of py-spark Lazy evaluation using Databricks.</p><p>What is Lazy evaluation in spark -</p><p>In <strong>PySpark, lazy evaluation</strong> means Spark <strong>does not execute transformations immediately</strong>. Instead, it builds a plan of what needs to be done and executes it only when you call an <strong>action</strong>.</p><p>Let us go through the example -</p><p>In the below code block we have done transformation but no action so we cant see the final data!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JX6urKAdheO5SHUBJ0ES8A.png" /></figure><p>Now we will add the Action as display() function -</p><p>Now you can see the Action is called and Data is displayed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5FlzPvpa7IUZP7ezZxcIaA.png" /></figure><p>Transformations vs Actions-&gt;</p><p>Transformations → lazy.</p><p>Some examples of narrow and wide transformations -</p><blockquote>filter()<br>select()<br>withColumn()<br>groupBy()<br>join()<br>map()</blockquote><p>Some examples of Actions -</p><blockquote>show()<br>count()<br>collect()<br>first()<br>write.csv(…)</blockquote><h3>Why is lazy evaluation useful?</h3><p>Spark can <strong>optimize the entire plan before executing it</strong>.</p><p>For example:</p><p>df.filter(df.age &gt; 18) \<br> .select(“name”) \<br> .show()</p><p>Spark’s optimizer can determine an efficient way to perform these operations rather than blindly executing each line separately.</p><p><strong>Simple way to remember:</strong></p><blockquote><strong><em>Transformation = “What should Spark do?”</em></strong><em><br> </em><strong><em>Action = “Okay, Spark, do it now.”</em></strong></blockquote><p>Hope you have enjoyed this short py-spark article on Lazy Evaluation!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=649f13ca346c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DataBricks AI Functions ; Are you Ready for your Monday Morning ?]]></title>
            <link>https://medium.com/@khasnobis.sanjit890/databricks-ai-functions-are-you-ready-for-your-monday-morning-8c83c32c4442?source=rss-5bf4882d9b72------2</link>
            <guid isPermaLink="false">https://medium.com/p/8c83c32c4442</guid>
            <dc:creator><![CDATA[Sanjit Khasnobis]]></dc:creator>
            <pubDate>Sun, 20 Sep 2026 09:27:54 GMT</pubDate>
            <atom:updated>2026-09-20T09:27:54.060Z</atom:updated>
            <content:encoded><![CDATA[<p>While the world is moving at lightning speed to automate workflows, we humans still have one workflow that refuses to be automated: <strong>Monday morning.</strong> 😂</p><p>Every Monday, we wake up feeling a little <em>Monday Blue</em>, trying to find the energy to start a brand-new week.</p><p>Meanwhile, our AI Agent friends are already online with the <strong>same enthusiasm, energy, and productivity</strong> they had when you said goodbye to them on Friday evening — right before you disappeared to adjust your Friday-night party outfit. 🕺💃🤖</p><p>No weekend hangover.<br>No Monday blues.<br>No “5 more minutes, please.”<br>Just pure, unstoppable enthusiasm.</p><p>Though AI can be scary at times, I still choose to appreciate the incredible advancements it is creating — especially when it comes to solving increasingly complex software engineering challenges.</p><p>There’s certainly a lot to learn, adapt to, and sometimes even worry about. But watching AI take on problems that once seemed incredibly difficult is also pretty exciting.</p><p>Today I will shortly discuss beauty of AI Functions in Databricks.</p><p>Databricks provides the following common functions, in case you want to take a quick look and check whether you’ve at least heard of them — or have a basic understanding of what each one does.</p><p>A quick glance might be all you need to refresh your memory or discover something new!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oO7sr4aP8p0a9cohjLQQOQ.png" /></figure><p>Enough of this lazy writing! 😄</p><p>Let’s jump into our <strong>favorite playground: code</strong> — the place where we humans have spent years proudly beating AI Agents at their own game.</p><p>But this time, there’s a twist…</p><p><strong>We’re going to beat the AI Agents… using AI Agents.</strong> 🤖⚔️🤖</p><p>Yep, things are getting interesting. Let the battle of the bots begin! 🚀</p><p>For this exercise I have took one sample pdf file about furniture installation where there is installation steps, guidance and images about the furniture parts, assembly parts.</p><p>For this exercise you can use DataBricks 14 days trial 400$ free credit option.</p><p>The pdf file looks like below -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OgPDU6Gzobv0bvfFKPaU8Q.png" /></figure><p>So here my targets are -</p><ol><li>To put sample pdf file in Databricks Volume and process it by DataBricks AI function and create a sample table in Databricks.</li><li>Query Databricks for some sample question.</li></ol><p>I have 2 input file</p><ol><li>One file as pdf where there is a details for one furniture</li><li>One csv file which have data for other products (assume already processed by AI functions before)</li></ol><p>The producr.csv file looks like this as below -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*77Y8_M9NaXO_Y0tqqpWvPg.png" /></figure><p>Now I will Use below Code piece to -</p><ol><li>Create Databricks Volume</li><li>Upload pdf and csv files there</li></ol><blockquote>import os</blockquote><blockquote>import shutil</blockquote><blockquote># Configuration (used by all following cells)</blockquote><blockquote>catalog = “furniture_catalog”</blockquote><blockquote>schema = “docs”</blockquote><blockquote>volume = “assembly_docs”</blockquote><blockquote>file_name = “The ExpanTable.pdf”</blockquote><blockquote>volume_path = f”/Volumes/{catalog}/{schema}/{volume}/manuals”</blockquote><blockquote>sample_pdf = f”{volume_path}/{file_name}”</blockquote><blockquote>images_path = f”/Volumes/{catalog}/{schema}/{volume}/extracted_images/”</blockquote><blockquote># 0. Create the Unity Catalog if it does not exist</blockquote><blockquote><strong>#spark.sql(f”CREATE CATALOG IF NOT EXISTS {catalog}”)</strong></blockquote><blockquote># 1. Create the Unity Catalog volume</blockquote><blockquote><strong>spark.sql(f”CREATE SCHEMA IF NOT EXISTS {catalog}.{schema}”)</strong></blockquote><blockquote><strong>spark.sql(f”CREATE VOLUME IF NOT EXISTS {catalog}.{schema}.{volume}”)</strong></blockquote><blockquote># Get current notebook’s directory, then navigate to repo root</blockquote><blockquote>current_dir = os.getcwd()</blockquote><blockquote>repo_root = os.path.dirname(current_dir)</blockquote><blockquote>data_path = os.path.join(repo_root, “data”, “assembly-docs”)</blockquote><blockquote># 2. Copy the products_silver CSV to the volume for later use</blockquote><blockquote>csv_source = os.path.join(repo_root, “data”,”products_silver.csv”)</blockquote><blockquote>csv_volume_path = f”/Volumes/{catalog}/{schema}/{volume}/products_silver.csv”</blockquote><blockquote>try:</blockquote><blockquote>shutil.copy2(csv_source, csv_volume_path)</blockquote><blockquote>print(f”✓ Copied: products_silver.csv to volume”)</blockquote><blockquote>except Exception as e:</blockquote><blockquote>print(f”✗ Failed to copy products_silver.csv: {e}”)</blockquote><blockquote># 3. Copy each PDF from repo (Workspace) into the volume using Python file operations</blockquote><blockquote>print(f”Copying PDFs from: {data_path}”)</blockquote><blockquote>print(f”To volume path: {volume_path}”)</blockquote><blockquote># Create the manuals directory in the volume if it doesn’t exist</blockquote><blockquote>os.makedirs(volume_path, exist_ok=True)</blockquote><blockquote># Get list of PDF files</blockquote><blockquote>pdf_files = [f for f in os.listdir(data_path) if f.endswith(“.pdf”)]</blockquote><blockquote>print(f”Found {len(pdf_files)} PDF files”)</blockquote><blockquote>for pdf_file in pdf_files:</blockquote><blockquote>source = os.path.join(data_path, pdf_file)</blockquote><blockquote>dest = os.path.join(volume_path, pdf_file)</blockquote><blockquote>try:</blockquote><blockquote>shutil.copy2(source, dest)</blockquote><blockquote>print(f”✓ Copied: {pdf_file}”)</blockquote><blockquote>except Exception as e:</blockquote><blockquote>print(f”✗ Failed to copy {pdf_file}: {e}”)</blockquote><blockquote>print(f”\nCreated volume {catalog}.{schema}.{volume} and copied {len(pdf_files)} PDF(s) to {volume_path}/”)</blockquote><p>So when I run my above code in Databricks workspace it created the require Catalog and volume and then push the files to the required location -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gBWb22HikWKcTuEez07HaA.png" /></figure><p>Here is a glimpse of my copied files -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6ywPmWNAEhERhmG8Kfb-mA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q56zviTLWSnD-KnifGUyVg.png" /></figure><p>Now fun part I will parse teh document using AI Function and will try to create a table -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OiN1LQ-jPt_71stBSClXsg.png" /></figure><p>Whats happening here — <strong>READ_FILES</strong> reads the binary file. <strong>ai_parse_document</strong> is a <em>task-specific</em> AI function that extracts structured content from unstructured documents using a state-of-the-art generative AI model hosted by Databricks. The result (text and image descriptions) is stored in a table so we can query it with SQL.</p><p>Ok lets inspect what it stored in table —</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j-LkeeHVt86VKxqyISDkfw.png" /><figcaption>One row per document. The parsed_content column holds the structured output (e.g. JSON with pages and text) that we’ll pass into the AI functions</figcaption></figure><p>Now, let’s fire off a quick question using the <strong>AI_Gen AI function</strong> and let Databricks deal with your <strong>insanely complicated query</strong>. 🤖🔥</p><p>Because apparently, asking AI to answer questions is now easier than asking a human who actually knows the answer. 😂</p><p><strong>Let’s see what Databricks comes up with!</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m-nzh79642s2yu1Tp1dSJw.png" /></figure><p>Wow its correct its said 2 people required.</p><p>So we have parsed and the query answer from pdf file using Databricks AI function by very few lines of code No heavy parsing No metadata management No Error handling!</p><p>Welcome to the world of LLM!</p><p>Now we will use AI_Query AI function where I will specify a LLM model name to do job for me as below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rAmbyAPP-6kv6jJJISV9Ug.png" /></figure><p>Watch carefully I have used <strong>meta-llama-3–3–70b-instruct</strong> model to answer my query from the pdf of furniture doc.</p><p>Question : What hardware parts come with this furniture? List them clearly and their quantities.</p><p>Answer given by Databricks —</p><p>Based on the provided JSON data, the hardware parts that come with the furniture are:</p><ol><li>M8 x 19mm bolts (quantity: 8)</li><li>2. M8 bolts (quantity: not specified, but mentioned multiple times)</li></ol><blockquote><strong>Note that the quantity of M8 bolts is not explicitly stated in the provided data, but it is mentioned in several places, suggesting that it is a required hardware part. The quantity of M8 x 19mm bolts is specified as 8.</strong></blockquote><p>Just see below picture you will understnad how clever LLM is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3bJ-YGLK6nZSu-jR70mGZQ.png" /></figure><p>Hope you have enjoyed this article of Databricks!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8c83c32c4442" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Databricks Volume Load Data and Create table using read_files()!]]></title>
            <link>https://medium.com/@khasnobis.sanjit890/databricks-volume-load-data-and-create-table-using-read-files-089796144807?source=rss-5bf4882d9b72------2</link>
            <guid isPermaLink="false">https://medium.com/p/089796144807</guid>
            <category><![CDATA[databricks-volumes]]></category>
            <category><![CDATA[read-file]]></category>
            <dc:creator><![CDATA[Sanjit Khasnobis]]></dc:creator>
            <pubDate>Thu, 17 Sep 2026 15:07:29 GMT</pubDate>
            <atom:updated>2026-09-17T15:07:29.400Z</atom:updated>
            <content:encoded><![CDATA[<p>Today we will try to do 2 steps -</p><ol><li>Load the Data to Databricks Volume</li><li>Create a table from the Data loaded in Databrick Volume.</li></ol><p>Let us first create a Databricks Volume as below -</p><p>Go To Catalog- Under Catalog go to Schema in our case we will go to my_retail_schema as below. If we go there and click on create we can see Volume is coming in the list.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xGZaK8ooJGetVdR-cRXc3w.png" /></figure><p>Very good now create a Volume name it as my_retail_volume.</p><p>I am going to create a managed Volume as below -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0Xx4vII_YKBxmdow_hbMNA.png" /></figure><p>Now you can see volume is created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uXwjCjW99ZC9SKUbZQ_pRw.png" /></figure><p>Now you can click on upload to this volume button to upload files into volume.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8WP65taS1maaP7DMOliFeg.png" /></figure><p>We want to take Products.csv and Sales.csv.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lKfbHuKdswT0-KMgvmwqEA.png" /></figure><p>Now click on upload.</p><p>You can see below Databricks is trying to upload the files to Databricks Managed Volume.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N2xXq_OzqmPjmtNk6wfQPg.png" /></figure><p>Now you can see both the files are uploaded now in my_retail_volume volume as below -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M-1l_6d2wRaqEFzgSmaVsA.png" /></figure><p>Now let us try to query the data from Databricks Volume for that open a new notebook as below.</p><p>Click on new notebook as below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zJWegNEj-qyqQ1XHGen8FQ.png" /></figure><p>In the notebook you can try to write below query -</p><blockquote>select * from text.`/Volumes/databrickssanjit/my_retail_schema/my_retail_volume/Products_CSV.csv`</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_HW6Sw8G3-d8AGUSkuj5aw.png" /></figure><p>How can I see data column wise ? you can just replace text with csv as below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OGVSE_nRVP2pJE0YRvGSLQ.png" /></figure><p>Now let us try to create a table from the data from Volume as below</p><p>You can try to run below create table statement -</p><blockquote>CREATE OR REPLACE TABLE Products AS</blockquote><blockquote>SELECT</blockquote><blockquote>ProductID AS product_id,</blockquote><blockquote>`Product Name` AS product_name,</blockquote><blockquote>Category AS category,</blockquote><blockquote>SubCategory AS subcategory,</blockquote><blockquote>`Unit Price (MRP)` AS unit_price_mrp</blockquote><blockquote>FROM <strong>read_files</strong>(</blockquote><blockquote>‘/Volumes/databrickssanjit/my_retail_schema/my_retail_volume/Products_CSV.csv’,</blockquote><blockquote>format =&gt; ‘csv’,</blockquote><blockquote>header =&gt; true</blockquote><blockquote>);</blockquote><p>The table should be cretaed without error as below -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ii5fW5nKLqRbnSyP5cGHhA.png" /></figure><p>Now let us query the data</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G5dy14abP8zluLeDnEjyjA.png" /></figure><p>Good Job!</p><p>So first you have created DataBricks Volume.</p><p>Then you have uploaded Products_CSV.csv and Sales_CSV.csv in Databricks Volume.</p><p>Then you have created a table from Products_CSV.csv data files in Volume.</p><p>And finally you query the data from Databricks Notebook!</p><p>Hope you have enjoyed the Article!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=089796144807" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DataBricks — Create Delta table and Run sql query on it!]]></title>
            <link>https://medium.com/@khasnobis.sanjit890/databricks-create-delta-table-and-run-sql-query-on-it-3632b4497aac?source=rss-5bf4882d9b72------2</link>
            <guid isPermaLink="false">https://medium.com/p/3632b4497aac</guid>
            <dc:creator><![CDATA[Sanjit Khasnobis]]></dc:creator>
            <pubDate>Wed, 16 Sep 2026 14:35:10 GMT</pubDate>
            <atom:updated>2026-09-16T14:35:10.880Z</atom:updated>
            <content:encoded><![CDATA[<p>Today we will go through how to create a Databricks table using csv file and then query the table from Databricks workspace.</p><p>First log into DataBricks workspace as below -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2LWCx2LLx-1wSI1cg_Qb1g.png" /></figure><p>I have already created a Workspace named as databrickssanjit.</p><p>Now I will create a new schema on it named as my_retail_schema.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*surz0pX-BeF0BZr1Ios-dQ.png" /></figure><p>You can see the my_retail_schema is created!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zjve4pG_Q1sD7SRaEYvmTA.png" /></figure><p>Now I will try to import my customer.csv file to the schema as a delta table.</p><p>Create on Schema and click on Create table as below -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dU0gYq4aQmbquX8Mcgw7-w.png" /></figure><p>It will take you to the next sceen as below -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9nwbRtkosfxSktj2QPQTww.png" /></figure><p>Browse to the machine location you have your customer.csv file as below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j_-h2qY4Bjq48EOi4gCG5w.png" /></figure><p>Now you have to click on Creat table as in below screen -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mq74qZL_eaQOwjhKGaJqSQ.png" /></figure><p>You can see your table is getting created as below-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WFoUSw5waynrYcGNj1VW8Q.png" /></figure><p>Your customer table is created as below -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LvlEDayfTe7ZyAfEoDO-Pw.png" /></figure><p>You can see it is a managed table and delta table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2lszbWihkI5iWjAX_ZRsBA.png" /></figure><p>Now let us try to query the table -</p><p>Let us go to the sql editor as below -</p><p>you can see I have queried the data from delta table just now loaded in databricks sql editor!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*76EXBrlOoUeoSZ0Dfp-VKQ.png" /></figure><p>Hope you have enjoyed this article!</p><p>Please keep following more Data Engineering articles!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3632b4497aac" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Databricks Unity Catalog permission hierarchy]]></title>
            <link>https://medium.com/@khasnobis.sanjit890/databricks-unity-catalog-permission-hierarchy-aed2b6a4830c?source=rss-5bf4882d9b72------2</link>
            <guid isPermaLink="false">https://medium.com/p/aed2b6a4830c</guid>
            <dc:creator><![CDATA[Sanjit Khasnobis]]></dc:creator>
            <pubDate>Wed, 16 Sep 2026 12:05:39 GMT</pubDate>
            <atom:updated>2026-09-16T12:05:39.805Z</atom:updated>
            <content:encoded><![CDATA[<p>Databricks becoming favourite Data Platform for AI GenAI use cases I feel it is worth to revisit the DataBricks Unity Catalog permission Hiearchy as below -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/992/1*3JGWKVQfUC_USrbrQ6t5wg.png" /></figure><p>We can also put a sketch across this as below -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S5wtslDNFlI5mnZsn7PAVA.png" /></figure><p>At the top Level we have Databricks Account.</p><p>A Databricks Account can manage multiple <strong>Databricks workspaces</strong>. Unity Catalog provides centralized governance across workspaces through a <strong>Unity Catalog metastore</strong>, which can be attached to multiple workspaces.</p><p>Access to data is primarily governed through <strong>Unity Catalog’s three-level namespace</strong>: Catalog -&gt; Schema -&gt; objects.</p><p>Objects can be as below -</p><ul><li><strong>Table</strong> — Stores structured data in rows and columns and can be queried using SQL.</li><li><strong>View</strong> — A virtual table created from a SQL query; it does not normally store the underlying data itself.</li><li><strong>Volume</strong> — A governed storage location in Unity Catalog for storing and accessing files.</li><li><strong>Function</strong> — A reusable piece of logic that accepts inputs and returns a value.</li><li><strong>Model</strong> — A registered machine-learning model that can be managed, versioned, and used for inference.</li></ul><p>So when you have to access any object like table you will query like</p><p><strong>&lt;catalog_name&gt;.&lt;schema_name&gt;.&lt;object_name&gt;</strong></p><p>When working in a larger Data Engineering AI Engneering team we see lot of people get confused about this simple fundamentals.</p><p>My suggestion is to go through this quick concept or redirect them to this link if you feel the same way!</p><p>Happy Learning! Hapy Coding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aed2b6a4830c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Snowflake Cortex — Complete Function, Task Specific Function and Helper Function -]]></title>
            <link>https://medium.com/@khasnobis.sanjit890/snowflake-cortex-complete-function-task-specific-function-and-helper-function-9f400412fbb2?source=rss-5bf4882d9b72------2</link>
            <guid isPermaLink="false">https://medium.com/p/9f400412fbb2</guid>
            <dc:creator><![CDATA[Sanjit Khasnobis]]></dc:creator>
            <pubDate>Sat, 12 Sep 2026 15:15:49 GMT</pubDate>
            <atom:updated>2026-09-12T15:15:49.240Z</atom:updated>
            <content:encoded><![CDATA[<h3>Snowflake Cortex AI Functions: Turn SQL Into an AI Data Processing Layer</h3><p><strong>Hook:</strong><br> Most enterprise data is not sitting neatly inside structured columns.</p><p>It lives in customer tickets, contracts, emails, documents, product reviews, call recordings, images, and other unstructured content.</p><p>Traditionally, getting business value from that data meant moving it into separate AI pipelines.</p><p><strong>Snowflake Cortex AI Functions change that model.</strong></p><p>Ask me How?</p><p>You can bring AI-powered <strong>extraction, classification, summarization, translation, sentiment analysis, embeddings, document parsing, and generation</strong> directly into Snowflake workflows using SQL.</p><h3>Think of Cortex AI Functions as an AI toolkit inside SQL</h3><p>Instead of building a separate pipeline for every task:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*5iAqxyivMQgscOjxKmUrng.png" /></figure><p>Snowflake also provides helper functions such as AI_COUNT_TOKENS, TO_FILE, and PROMPT to support AI workflows.</p><h3>Example: turn customer tickets into actionable data</h3><p>Imagine a table containing thousands of support tickets:</p><pre>SELECT<br>    ticket_id,<br>    AI_CLASSIFY(<br>        ticket_text,<br>        [&#39;Billing&#39;, &#39;Technical&#39;, &#39;Account&#39;, &#39;Product Feedback&#39;]<br>    ) AS category,<br>    AI_SENTIMENT(ticket_text) AS sentiment,<br>    AI_COMPLETE(<br>        &#39;Summarize this customer issue in one sentence: &#39; || ticket_text<br>    ) AS summary<br>FROM support_tickets;</pre><p>Now the same dataset can contain:</p><p><strong>Raw ticket → category → sentiment → summary</strong></p><p>without creating a separate application just to perform those transformations.</p><h3>And it goes beyond text</h3><p>Cortex AI Functions can work with images and documents as well.</p><p>For example, AI_PARSE_DOCUMENT can extract text and layout information from documents, while AI_EXTRACT can turn information from documents and other inputs into structured data.</p><p>That opens up workflows such as:</p><p><strong>PDF invoices → extracted fields → structured table → analytics</strong></p><p>or:</p><p><strong>Contracts → extracted clauses → classification → compliance analysis</strong></p><h3>Where this becomes really interesting ??</h3><p>The biggest opportunity isn’t using one AI function.</p><p>It’s <strong>composing them</strong>.</p><p>For example:</p><pre>Document<br>   ↓<br>AI_PARSE_DOCUMENT<br>   ↓<br>AI_EXTRACT<br>   ↓<br>AI_CLASSIFY<br>   ↓<br>AI_REDACT<br>   ↓<br>Analytics / RAG / downstream application</pre><p>This turns Snowflake from simply being the place where your structured data lives into a place where <strong>unstructured data can be transformed into analytics-ready information</strong>.</p><p>Snowflake specifically describes these capabilities as useful for document analytics, RAG pipelines, filtering, classification, sentiment analysis, translation, and entity extraction.</p><h3>The important architectural point</h3><p>Cortex AI Functions are particularly suited to <strong>batch processing at scale</strong>. Snowflake recommends them for processing numerous inputs, such as text across large SQL tables. For highly interactive, latency-sensitive inference, Snowflake points to its APIs instead.</p><p>So the mental model is:</p><p><strong>SQL + AI Functions → data transformation at scale</strong></p><p>rather than:</p><p><strong>Application → API call → one piece of content at a time</strong></p><h3>Bottom line</h3><p>Snowflake Cortex AI Functions aren’t just another collection of LLM wrappers.</p><p>Their interesting proposition is that <strong>AI operations become composable data operations</strong>.</p><p>You can take unstructured enterprise content, apply AI transformations to it, and feed the results directly into the same analytical workflows where your structured data already lives.</p><p>That makes the real question less:</p><blockquote>“How do I add an LLM to my data platform?”</blockquote><p>and more:</p><blockquote><strong><em>“Which parts of my data pipeline can now become AI-powered SQL?</em></strong></blockquote><p>So Snowflex Cortex is brining surely exciting power at the hand of Data Engineers , Data Scientists and Analytics Engineer!</p><p>Hope you have enjoyed the article and agreed on the above view points!</p><p>If not please share your viewpoint Love to Hear from you!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9f400412fbb2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Google ADK : High Level what is what?]]></title>
            <link>https://medium.com/@khasnobis.sanjit890/google-adk-high-level-what-is-what-ca00b7e1b71b?source=rss-5bf4882d9b72------2</link>
            <guid isPermaLink="false">https://medium.com/p/ca00b7e1b71b</guid>
            <category><![CDATA[google-adk]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <dc:creator><![CDATA[Sanjit Khasnobis]]></dc:creator>
            <pubDate>Sat, 08 Aug 2026 11:18:35 GMT</pubDate>
            <atom:updated>2026-08-08T11:18:35.538Z</atom:updated>
            <content:encoded><![CDATA[<p>Today, I want to touch upon a very interesting framework I recently came across while working on one of my projects.</p><p>As everyone is jumping into this new world of <strong>AI Engineering</strong>, I couldn’t help but wonder:</p><p><strong>Are we really entering an entirely new world, or are we going back to something we already know?</strong></p><p>When I look at modern AI application architectures, I see familiar concepts coming back — <strong>session management, memory management, context management, orchestration, tools, state, events, and distributed services.</strong></p><p>It feels less like we’re inventing everything from scratch and more like <strong>history repeating itself</strong>.</p><p>The difference?</p><p>We now have a remarkably talented new kid in the market — the <strong>almighty LLM</strong>. 😄</p><p>The LLM brings a new reasoning capability to the architecture, but around it, we’re once again building the engineering foundations we’ve been building for decades.</p><p>Maybe <strong>AI Engineering isn’t replacing traditional software engineering.</strong></p><p>Maybe it’s <strong>traditional software engineering evolving around a new reasoning engine.</strong></p><p>And that’s exactly what makes frameworks like <strong>Google ADK</strong> so interesting to me.</p><p>Lets try to decode google adk a bit in todays article!</p><p><strong>Google ADK (Agent Development Kit)</strong> as a runtime/framework for building agents where the <strong>Agent</strong> contains the reasoning logic, <strong>Tools</strong> let it interact with the outside world, and the <strong>Runner + Session + Services</strong> manage execution and state.</p><h3>1. Complete Google ADK architecture</h3><pre>┌──────────────────────────────┐<br>                         │          USER / APP           │<br>                         │ Web / Mobile / API / Chat UI  │<br>                         └──────────────┬───────────────┘<br>                                        │<br>                                        │ User message<br>                                        ▼<br>                         ┌──────────────────────────────┐<br>                         │       ADK API / Server        │<br>                         │   FastAPI / ADK Web / A2A     │<br>                         └──────────────┬───────────────┘<br>                                        │<br>                                        ▼<br>┌───────────────────────────────────────────────────────────────────────────┐<br>│                           ADK RUNTIME                                     │<br>│                                                                           │<br>│  ┌─────────────────────┐          ┌──────────────────────────────────┐   │<br>│  │      Runner         │─────────▶│             Session              │   │<br>│  │                     │          │                                  │   │<br>│  │ • execution loop    │          │ • conversation history           │   │<br>│  │ • event processing  │          │ • state                           │   │<br>│  │ • agent invocation  │          │ • user/session/app identifiers   │   │<br>│  └──────────┬──────────┘          └──────────────────────────────────┘   │<br>│             │                                                             │<br>│             ▼                                                             │<br>│  ┌────────────────────────────────────────────────────────────────────┐  │<br>│  │                         ROOT AGENT                                  │  │<br>│  │                                                                    │  │<br>│  │  Name + Instructions + Model + Tools + Sub-agents + Callbacks      │  │<br>│  │                                                                    │  │<br>│  │       ┌──────────────┐    ┌──────────────┐    ┌──────────────┐     │  │<br>│  │       │   LLM Model  │    │    Tools     │    │ Sub-Agents   │     │  │<br>│  │       │              │    │              │    │              │     │  │<br>│  │       │ Gemini       │    │ Python       │    │ Sequential   │     │  │<br>│  │       │ Vertex AI    │    │ MCP          │    │ Parallel     │     │  │<br>│  │       │ Other models │    │ API          │    │ Loop         │     │  │<br>│  │       └──────┬───────┘    └──────┬───────┘    └──────┬───────┘     │  │<br>│  │              │                   │                   │             │  │<br>│  └──────────────┼───────────────────┼───────────────────┼─────────────┘  │<br>│                 │                   │                   │                │<br>│                 ▼                   ▼                   ▼                │<br>│          ┌────────────┐     ┌──────────────┐    ┌────────────────┐      │<br>│          │  LLM Call  │     │ External     │    │ Other Agents   │      │<br>│          │            │     │ Systems      │    │ / A2A          │      │<br>│          └────────────┘     └──────────────┘    └────────────────┘      │<br>│                                                                           │<br>│  ┌────────────────────────────────────────────────────────────────────┐  │<br>│  │                       ADK SERVICES                                  │  │<br>│  │                                                                    │  │<br>│  │ Session Service │ Memory Service │ Artifact Service │ Credential   │  │<br>│  │                                                                    │  │<br>│  └────────────────────────────────────────────────────────────────────┘  │<br>│                                                                           │<br>│  ┌────────────────────────────────────────────────────────────────────┐  │<br>│  │                  CALLBACKS / PLUGINS / EVENTS                      │  │<br>│  │                                                                    │  │<br>│  │ before_agent │ after_agent │ before_model │ after_model             │  │<br>│  │ before_tool  │ after_tool  │ event processing / telemetry           │  │<br>│  └────────────────────────────────────────────────────────────────────┘  │<br>└───────────────────────────────────────────────────────────────────────────┘<br>                                        │<br>                                        ▼<br>                         ┌──────────────────────────────┐<br>                         │       EXTERNAL SYSTEMS       │<br>                         │                              │<br>                         │ Vertex AI / Gemini           │<br>                         │ Databases                    │<br>                         │ REST APIs                    │<br>                         │ MCP servers                  │<br>                         │ Google Search                │<br>                         │ Cloud Storage                 │<br>                         │ Enterprise systems            │<br>                         └──────────────────────────────┘</pre><h3>2. The most important ADK components</h3><p>There are roughly <strong>10 concepts</strong> you should understand.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1007/1*265Fpxyj_3PiDSCgkqJ90g.png" /></figure><p>The current Google tooling also supports deployment through <strong>Agent Runtime, Cloud Run, or GKE</strong>, while ADK agents can expose <strong>A2A</strong> for agent-to-agent interoperability.</p><h3>3. Agent</h3><p>The <strong>Agent</strong> is the central building block.</p><p>Conceptually:</p><pre>root_agent = Agent(<br>    name=&quot;customer_support&quot;,<br>    model=&quot;gemini-...&quot;,<br>    instruction=&quot;Help customers solve their problems.&quot;,<br>    tools=[<br>        search_customer,<br>        create_ticket,<br>        check_order<br>    ]<br>)</pre><p>An agent normally defines:</p><pre>Agent<br> ├── name<br> ├── model<br> ├── instruction<br> ├── tools<br> ├── sub_agents<br> ├── callbacks<br> └── configuration</pre><p>So:</p><pre>Agent<br>                      │<br>        ┌─────────────┼──────────────┐<br>        │             │              │<br>      Model          Tools        Sub-agents<br>        │             │              │<br>      Gemini       APIs/MCP       Specialists</pre><h3>4. Model</h3><p>The model is the <strong>reasoning engine</strong>.</p><p>For example:</p><pre>Agent<br>                   │<br>                   ▼<br>             ┌───────────┐<br>             │   Model   │<br>             └─────┬─────┘<br>                   │<br>           ┌───────┴────────┐<br>           │                │<br>        Gemini          Other model</pre><p>With Google’s ecosystem, Gemini/Vertex AI is the common path.</p><p>The model receives context assembled by the runtime and may decide:</p><pre>User request<br>     │<br>     ▼<br>    LLM<br>     │<br>     ├── Answer directly<br>     │<br>     ├── Call tool<br>     │<br>     └── Delegate to another agent</pre><h3>5. Tools</h3><p><strong>Tools are how the agent interacts with the real world.</strong></p><p>For example:</p><pre>Agent<br>                   │<br>                   ▼<br>                 Model<br>                   │<br>             &quot;I need data&quot;<br>                   │<br>                   ▼<br>                 Tool<br>                   │<br>        ┌──────────┼──────────┐<br>        ▼          ▼          ▼<br>      REST       Database     MCP<br>       API        query      server</pre><p>Typical tools:</p><pre>Python function<br>REST API<br>Database<br>Google Search<br>MCP tool<br>File system<br>Cloud service<br>Custom enterprise API</pre><p>Example:</p><pre>def get_order(order_id: str):<br>    return database.find_order(order_id)</pre><p>The model can decide:</p><pre>&quot;I need order information&quot;</pre><pre>        ↓</pre><pre>get_order(&quot;12345&quot;)</pre><pre>        ↓</pre><pre>database</pre><pre>        ↓</pre><pre>order result</pre><pre>        ↓</pre><pre>LLM continues reasoning</pre><p>This is one of the most important concepts in agent architecture:</p><blockquote><strong><em>The LLM decides what to do; the tool actually does it.</em></strong></blockquote><h3>6. Runner</h3><p>The <strong>Runner</strong> is responsible for executing the agent.</p><p>This is different from the Agent itself.</p><p>Think:</p><pre>Agent = WHAT the agent is</pre><pre>Runner = HOW the agent runs</pre><p>Architecture:</p><pre>User message<br>                   │<br>                   ▼<br>              ┌────────┐<br>              │ Runner │<br>              └───┬────┘<br>                  │<br>                  ▼<br>               Agent<br>                  │<br>           ┌──────┴──────┐<br>           ▼             ▼<br>         Model          Tools</pre><p>The Runner manages the execution lifecycle and events.</p><p>A simplified execution loop looks like:</p><pre>User input<br>    │<br>    ▼<br>Runner<br>    │<br>    ▼<br>Agent<br>    │<br>    ▼<br>Model<br>    │<br>    ├──────► final answer<br>    │<br>    ├──────► tool call<br>    │             │<br>    │             ▼<br>    │           Tool<br>    │             │<br>    │             ▼<br>    │          Tool result<br>    │             │<br>    │             ▼<br>    │           Model<br>    │<br>    └──────► another agent</pre><h3>7. Session</h3><p>A <strong>Session</strong> represents a particular interaction/conversation.</p><p>For example:</p><pre>User: Alice</pre><pre>Session ID:<br>abc123</pre><pre>History:<br>────────────────────────────<br>User: What&#39;s my order?<br>Agent: Which order?</pre><pre>User: Order 123<br>Agent: It shipped yesterday.</pre><pre>User: Where is it now?<br>Agent: ...<br>────────────────────────────</pre><p>A session typically contains:</p><pre>Session<br> ├── session_id<br> ├── app/user identifiers<br> ├── conversation events<br> ├── state<br> └── other execution information</pre><p>Important distinction:</p><h3>Session ≠ Memory</h3><p>Session:</p><blockquote><em>“What happened in this conversation?”</em></blockquote><p>Memory:</p><blockquote><em>“What useful information should I remember across conversations?”</em></blockquote><h3>8. Session Service</h3><p>The <strong>Session Service</strong> manages sessions.</p><p>Conceptually:</p><pre>Runner<br>                    │<br>                    ▼<br>            Session Service<br>                    │<br>        ┌───────────┼────────────┐<br>        ▼           ▼            ▼<br>    In-memory     Cloud SQL    Agent Platform<br>                              Sessions</pre><p>For development, you can use in-memory storage.</p><p>For production, persistent session storage is normally appropriate.</p><p>The current Google tooling exposes session-storage choices including <strong>in-memory, Cloud SQL, and Agent Platform Sessions</strong>.</p><h3>9. State</h3><p>State is different from conversation history.</p><p>Example:</p><pre>state = {<br>    &quot;customer_id&quot;: &quot;C123&quot;,<br>    &quot;language&quot;: &quot;en&quot;,<br>    &quot;cart_total&quot;: 125.50,<br>    &quot;authenticated&quot;: True<br>}</pre><p>Think:</p><pre>Session<br>│<br>├── Events / conversation<br>│<br>└── State<br>      ├── user preferences<br>      ├── temporary values<br>      ├── workflow values<br>      └── application state</pre><p>State is especially useful in multi-step workflows.</p><p>For example:</p><pre>Step 1:<br>collect customer information</pre><pre>        ↓</pre><pre>state.customer = ...</pre><pre>        ↓</pre><pre>Step 2:<br>verify customer</pre><pre>        ↓</pre><pre>state.customer_verified = true</pre><pre>        ↓</pre><pre>Step 3:<br>process request</pre><h3>10. Memory</h3><p>Memory is for <strong>longer-term information retrieval</strong>.</p><p>For example:</p><pre>Conversation 1<br>────────────────<br>User likes Python<br>User works at ABC<br>User prefers concise answers</pre><pre>                  ↓<br>               MEMORY<br>                  ↓<br>Conversation 50<br>────────────────<br>Agent retrieves relevant memories</pre><p>So:</p><pre>┌──────────────┐<br>             │    Agent     │<br>             └──────┬───────┘<br>                    │<br>             retrieve memory<br>                    │<br>                    ▼<br>             ┌──────────────┐<br>             │ Memory       │<br>             │ Service      │<br>             └──────────────┘</pre><p>This is different from RAG.</p><h3>RAG</h3><pre>Question<br>   ↓<br>Retriever<br>   ↓<br>Documents<br>   ↓<br>LLM</pre><h3>Agent memory</h3><pre>User/conversation history<br>          ↓<br>       Memory<br>          ↓<br>Relevant previous information<br>          ↓<br>        Agent</pre><p>You can also combine them:</p><pre>Agent<br>                   │<br>        ┌──────────┼──────────┐<br>        ▼          ▼          ▼<br>     Session     Memory      RAG<br>     history     service     system</pre><h3>11. Artifacts</h3><p>Artifacts are for <strong>files and binary data</strong>.</p><p>Examples:</p><pre>PDF<br>Image<br>Audio<br>Video<br>CSV<br>Generated report<br>Charts<br>Documents</pre><p>Architecture:</p><pre>Agent<br>  │<br>  ├── text → normal response<br>  │<br>  └── file → Artifact Service<br>                   │<br>                   ▼<br>             Artifact storage</pre><p>For example:</p><pre>User:<br>&quot;Analyze this CSV and generate a report.&quot;</pre><pre>        ↓</pre><pre>Agent<br>        ↓<br>Tool reads CSV<br>        ↓<br>Model analyzes data<br>        ↓<br>Generate report<br>        ↓<br>Artifact<br>        ↓<br>User downloads report</pre><h3>12. Events</h3><p>ADK execution can be thought of as an <strong>event stream</strong>.</p><p>For example:</p><pre>Event 1<br>User message</pre><pre>Event 2<br>Agent started</pre><pre>Event 3<br>Model request</pre><pre>Event 4<br>Model response</pre><pre>Event 5<br>Tool call</pre><pre>Event 6<br>Tool response</pre><pre>Event 7<br>Model response</pre><pre>Event 8<br>Final agent response</pre><p>This is important for:</p><ul><li>streaming</li><li>debugging</li><li>tracing</li><li>observability</li><li>UI updates</li><li>evaluation</li></ul><p>Conceptually:</p><pre>Runner<br>                      │<br>                      ▼<br>               Event stream<br>                      │<br>       ┌──────────────┼──────────────┐<br>       ▼              ▼              ▼<br>      UI           Logging        Tracing</pre><h3>13. Callbacks</h3><p>Callbacks let you intercept execution.</p><p>Typical lifecycle:</p><pre>Agent<br>                   │<br>          before_agent_callback<br>                   │<br>                   ▼<br>                Agent<br>                   │<br>          before_model_callback<br>                   │<br>                   ▼<br>                 Model<br>                   │<br>          after_model_callback<br>                   │<br>                   ▼<br>                Agent<br>                   │<br>          before_tool_callback<br>                   │<br>                   ▼<br>                 Tool<br>                   │<br>          after_tool_callback<br>                   │<br>                   ▼<br>                Agent<br>                   │<br>          after_agent_callback</pre><p>They are useful for:</p><ul><li>logging</li><li>authorization</li><li>validation</li><li>modifying requests</li><li>blocking dangerous operations</li><li>collecting metrics</li><li>injecting context</li><li>auditing</li></ul><h3>14. Multi-agent architecture</h3><p>This is where ADK becomes especially interesting.</p><p>You can have:</p><pre>Root Agent<br>                           │<br>          ┌────────────────┼────────────────┐<br>          │                │                │<br>          ▼                ▼                ▼<br>     Research Agent   Coding Agent     Finance Agent<br>          │                │                │<br>       Tools            Tools            Tools</pre><p>The root agent can delegate work.</p><p>For example:</p><pre>User<br> │<br> ▼<br>Travel Agent<br> │<br> ├──► Flight Agent<br> │<br> ├──► Hotel Agent<br> │<br> └──► Restaurant Agent</pre><h3>15. Sequential orchestration</h3><p>For workflows where steps happen in order:</p><pre>Root<br>              │<br>              ▼<br>       ┌──────────────┐<br>       │ Step 1 Agent │<br>       └──────┬───────┘<br>              ▼<br>       ┌──────────────┐<br>       │ Step 2 Agent │<br>       └──────┬───────┘<br>              ▼<br>       ┌──────────────┐<br>       │ Step 3 Agent │<br>       └──────┬───────┘<br>              ▼<br>            Result</pre><p>Example:</p><pre>Research<br>   ↓<br>Analyze<br>   ↓<br>Write<br>   ↓<br>Review</pre><h3>16. Parallel orchestration</h3><p>Independent tasks can execute concurrently.</p><pre>Root<br>                      │<br>             ┌────────┼────────┐<br>             ▼        ▼        ▼<br>          Search    Search    Search<br>          Agent     Agent     Agent<br>             │        │        │<br>             └────────┼────────┘<br>                      ▼<br>                  Aggregator<br>                      │<br>                      ▼<br>                    Answer</pre><p>For example:</p><pre>Research Agent<br>Finance Agent<br>Legal Agent<br>Technical Agent</pre><pre>       ↓</pre><pre>Synthesis Agent</pre><h3>17. Loop orchestration</h3><p>Useful when you need iterative refinement:</p><pre>┌───────────────┐<br>       │    Agent      │<br>       └───────┬───────┘<br>               │<br>               ▼<br>           Evaluate<br>               │<br>        ┌──────┴──────┐<br>        │             │<br>      Failed        Passed<br>        │             │<br>        ▼             ▼<br>      Retry          Done<br>        │<br>        └──────► Agent</pre><p>Example:</p><pre>Generate code<br>     ↓<br>Run tests<br>     ↓<br>Tests failed?<br>   /      \<br> yes      no<br>  ↓        ↓<br>Fix      Done<br>  │<br>  └──→ Run tests</pre><h3>18. A2A — Agent-to-Agent</h3><p>For distributed agents, ADK can expose agents through <strong>A2A</strong>.</p><pre>Agent A<br>              Coordinator<br>                   │<br>              A2A protocol<br>                   │<br>       ┌───────────┼───────────┐<br>       ▼           ▼           ▼<br>   Agent B      Agent C      Agent D<br>   Research     Finance      Coding</pre><p>This means the specialist agents don’t necessarily have to run inside the same process.</p><p>For example:</p><pre>Service 1<br>┌─────────────────┐<br>│ Research Agent  │<br>└─────────────────┘<br>        ▲<br>        │ A2A<br>        ▼<br>┌─────────────────┐<br>│ Coordinator     │<br>└─────────────────┘<br>        │ A2A<br>        ▼<br>┌─────────────────┐<br>│ Finance Agent   │<br>└─────────────────┘</pre><p>Current Google tooling describes A2A as built into the ADK template and supports distributed multi-agent systems.</p><h3>19. MCP integration</h3><p>ADK agents can also work with <strong>MCP (Model Context Protocol)</strong> servers.</p><p>Architecture:</p><pre>ADK Agent<br>                      │<br>                      ▼<br>                  ADK Model<br>                      │<br>                  Tool call<br>                      │<br>                      ▼<br>                  MCP Client<br>                      │<br>             ┌────────┼────────┐<br>             ▼        ▼        ▼<br>          MCP      MCP       MCP<br>         Server    Server    Server<br>          CRM       DB       GitHub</pre><p>So you can have:</p><pre>ADK<br> │<br> ├── Native ADK tools<br> │<br> ├── Python functions<br> │<br> ├── REST APIs<br> │<br> └── MCP tools</pre><h3>20. Complete runtime flow</h3><p>Let’s put everything together.</p><p>Suppose the user says:</p><blockquote><em>“Find my latest order and tell me when it will arrive.”</em></blockquote><p>The architecture is roughly:</p><pre>USER<br> │<br> │ &quot;Find my latest order...&quot;<br> ▼<br>API / ADK Server<br> │<br> ▼<br>RUNNER<br> │<br> ├──────────────► SESSION SERVICE<br> │                    │<br> │                    ▼<br> │               Load session<br> │<br> ▼<br>ROOT AGENT<br> │<br> ▼<br>CALLBACK<br> │<br> ▼<br>MODEL / GEMINI<br> │<br> │ decides:<br> │ &quot;I need order information&quot;<br> ▼<br>TOOL<br> │<br> │ get_latest_order()<br> ▼<br>ORDER API<br> │<br> ▼<br>Tool result<br> │<br> ▼<br>MODEL<br> │<br> │ determines answer<br> ▼<br>CALLBACK<br> │<br> ▼<br>AGENT<br> │<br> ▼<br>EVENT<br> │<br> ▼<br>SESSION SERVICE<br> │<br> │ persist conversation<br> ▼<br>RUNNER<br> │<br> ▼<br>API RESPONSE<br> │<br> ▼<br>USER</pre><h3>21. Production architecture</h3><p>For a production Google Cloud implementation, I’d structure it like this:</p><pre>INTERNET<br>                            │<br>                            ▼<br>                    ┌──────────────┐<br>                    │ API Gateway  │<br>                    │ / Load Bal.  │<br>                    └──────┬───────┘<br>                           │<br>                           ▼<br>                 ┌───────────────────┐<br>                 │   ADK Agent       │<br>                 │   Runtime         │<br>                 └─────────┬─────────┘<br>                           │<br>             ┌─────────────┼──────────────┐<br>             │             │              │<br>             ▼             ▼              ▼<br>         Root Agent      Session       Memory<br>             │            Service       Service<br>             │<br>      ┌──────┼─────────┐<br>      │      │         │<br>      ▼      ▼         ▼<br>    Model  Tools    Sub-agents<br>      │      │         │<br>      ▼      ▼         ▼<br>   Gemini   APIs       A2A<br>            │<br>     ┌──────┼───────────────┐<br>     ▼      ▼               ▼<br>   DB     MCP servers     GCP services<br></pre><pre>                  ┌─────────────────┐<br>                  │ Observability   │<br>                  │                 │<br>                  │ OpenTelemetry   │<br>                  │ Cloud Trace     │<br>                  │ Cloud Logging   │<br>                  │ BigQuery        │<br>                  └─────────────────┘</pre><p>Google’s current tooling supports deployment to <strong>Agent Runtime, Cloud Run, and GKE</strong>.</p><h3>22. ADK vs surrounding Google Cloud services</h3><p>This distinction is extremely important.</p><pre>┌───────────────────────────────────────────────────────────┐<br>│                    APPLICATION                            │<br>│                                                           │<br>│  UI / Mobile / Backend / API                              │<br>└──────────────────────────┬────────────────────────────────┘<br>                           │<br>                           ▼<br>┌───────────────────────────────────────────────────────────┐<br>│                         ADK                                │<br>│                                                           │<br>│  Agents                                                   │<br>│  Tools                                                    │<br>│  Model integration                                        │<br>│  Runner                                                   │<br>│  Sessions                                                 │<br>│  Memory                                                   │<br>│  Artifacts                                                │<br>│  Callbacks                                                │<br>│  Multi-agent orchestration                                │<br>└──────────────────────────┬────────────────────────────────┘<br>                           │<br>              ┌────────────┼────────────┐<br>              ▼            ▼            ▼<br>         Vertex AI      Databases      APIs<br>              │<br>              ▼<br>           Gemini</pre><p>ADK is therefore <strong>not the entire infrastructure</strong>.</p><p>It is primarily the <strong>agent development/runtime framework</strong>.</p><h3>23. Development → production architecture</h3><p>The overall lifecycle is:</p><pre>DEVELOPMENT<br>                     │<br>                     ▼<br>             ADK Agent Code<br>                     │<br>                     ▼<br>              Local Playground<br>                     │<br>                     ▼<br>                  Evaluate<br>                     │<br>                     ▼<br>              ┌──────────────┐<br>              │    Deploy    │<br>              └──────┬───────┘<br>                     │<br>          ┌──────────┼───────────┐<br>          ▼          ▼           ▼<br>     Agent Runtime Cloud Run    GKE<br>          │          │           │<br>          └──────────┼───────────┘<br>                     ▼<br>                Production<br>                     │<br>              ┌──────┴──────┐<br>              ▼             ▼<br>        Observability     Evaluation</pre><p>The current Google Agents CLI specifically supports the build → evaluate → deploy → observe lifecycle around ADK. (<a href="https://google.github.io/agents-cli/guide/getting-started/?utm_source=chatgpt.com">Google GitHub</a>)</p><h3>24. Observability</h3><p>For production, don’t think only about the final response.</p><p>You want to see:</p><pre>Request<br>  │<br>  ├── Agent execution<br>  │<br>  ├── LLM call<br>  │     ├── latency<br>  │     ├── tokens<br>  │     └── response<br>  │<br>  ├── Tool call<br>  │     ├── latency<br>  │     ├── arguments<br>  │     └── result<br>  │<br>  ├── Sub-agent<br>  │<br>  └── Final response</pre><p>Google’s current tooling uses Open Telemetry/Cloud Trace for distributed tracing, and can also capture prompt/response information into GCS and Big Query when configured.</p><h3>25. Evaluation</h3><p>A production agent should have an evaluation loop:</p><pre>Agent<br>                │<br>                ▼<br>           Test Dataset<br>                │<br>                ▼<br>          Run Agent<br>                │<br>                ▼<br>             Traces<br>                │<br>                ▼<br>           Evaluation<br>                │<br>       ┌────────┴────────┐<br>       ▼                 ▼<br>   Good result        Bad result<br>                         │<br>                         ▼<br>                  Improve prompt/<br>                  tools/architecture<br>                         │<br>                         └──────► Agent</pre><p>Current Google tooling supports evaluation workflows including generating traces, grading them, comparing results, analyzing failures, and experimental prompt optimization.</p><h3>26. The mental model I’d recommend</h3><p>If you’re learning ADK, remember this hierarchy:</p><pre>ADK<br>                          │<br>             ┌────────────┴────────────┐<br>             │                         │<br>          Runtime                   Agent<br>             │                         │<br>       ┌─────┼─────┐          ┌────────┼────────┐<br>       │     │     │          │        │        │<br>    Runner Session Events    Model    Tools   Subagents<br>            │<br>       ┌────┴─────┐<br>       │          │<br>     State      History</pre><pre>Services<br>   │<br>   ├── Session Service<br>   ├── Memory Service<br>   ├── Artifact Service<br>   └── Credential/Auth-related infrastructure</pre><pre>Integration<br>   │<br>   ├── MCP<br>   ├── A2A<br>   ├── REST APIs<br>   └── Google Cloud services</pre><pre>Production<br>   │<br>   ├── Agent Runtime<br>   ├── Cloud Run<br>   ├── GKE<br>   ├── Observability<br>   └── Evaluation</pre><h3>Below are key components you can keep a track when building agentic app using google adk -</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0vhA6EmNVcasREdAss8ZYQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QDtmRcSPxxC3IwffR_keWA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Du0tYpIOAhP4G6mZLGDryg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NbqHf6ydIJAckvRhSgO76A.png" /></figure><p>Hope you enjoyed the Article! A brief highlight on Google Agent Development Kit framework.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ca00b7e1b71b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Zero Copy Cloning in Snowflake]]></title>
            <link>https://medium.com/@khasnobis.sanjit890/zero-copy-cloning-in-snowflake-85618e4e1831?source=rss-5bf4882d9b72------2</link>
            <guid isPermaLink="false">https://medium.com/p/85618e4e1831</guid>
            <category><![CDATA[cloning]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[data-ware-housing]]></category>
            <dc:creator><![CDATA[Sanjit Khasnobis]]></dc:creator>
            <pubDate>Sat, 08 Aug 2026 10:18:32 GMT</pubDate>
            <atom:updated>2026-08-08T10:18:32.823Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vxvuizu37yuIqYa675St6w.png" /></figure><p>Today I am writing a very interesting topic in Snowflake. It is called Zero Copy Cloning.</p><p>In Snowflake, <strong>zero-copy cloning</strong> means creating a clone of a table without physically copying its existing data. As shown in the example, when SALES_CLONE is created from SALES, both tables initially reference the same micro-partitions (MP1, MP2, and MP3), so no additional storage is required for those partitions. The clone behaves like an independent table, but the underlying micro-partitions are shared. If you later modify the clone, Snowflake does not change the existing micro-partitions because they are immutable; instead, it creates new micro-partitions (such as MP4) containing the changed data. Thus, zero-copy cloning saves storage and is created almost instantly, while additional storage is consumed only as the cloned table changes.</p><h3>1. Clone is created — micro-partitions are shared</h3><pre>SOURCE TABLE<br>              ┌───────────────┐<br>              │  SALES        │<br>              └───────┬───────┘<br>                      │<br>                      │ references<br>                      ▼<br>             ┌─────────────────┐<br>             │ Micro-partition │<br>             │      MP1        │<br>             ├─────────────────┤<br>             │ Micro-partition │<br>             │      MP2        │<br>             ├─────────────────┤<br>             │ Micro-partition │<br>             │      MP3        │<br>             └────────┬────────┘<br>                      ▲<br>                      │ references<br>                      │<br>              ┌───────┴───────┐<br>              │  SALES_CLONE  │<br>              └───────────────┘</pre><pre>        CREATE TABLE SALES_CLONE<br>        CLONE SALES;</pre><pre>        No physical data is copied.<br>        Only metadata/references are created.</pre><p>The important idea is that <strong>MP1, MP2 and MP3 physically exist only once</strong>, but both tables can reference them.</p><h3>2. Clone is changed — new micro-partitions are created</h3><pre>SOURCE TABLE<br>           ┌───────────────┐<br>           │    SALES      │<br>           └───────┬───────┘<br>                   │<br>                   ▼<br>          ┌─────────────────┐<br>          │      MP1        │◄─────────┐<br>          ├─────────────────┤          │<br>          │      MP2        │◄──────┐  │<br>          ├─────────────────┤       │  │<br>          │      MP3        │◄───┐  │  │<br>          └─────────────────┘    │  │  │<br>                                 │  │  │<br>                           shared│  │  │<br>                                 │  │  │<br>                       ┌─────────┴──┴──┴──┐<br>                       │   SALES_CLONE    │<br>                       └─────────┬────────┘<br>                                 │<br>                          UPDATE / INSERT<br>                                 │<br>                                 ▼<br>                        ┌─────────────────┐<br>                        │      MP4        │<br>                        │  NEW partition  │<br>                        └─────────────────┘<br>                                 ▲<br>                                 │<br>                         clone references MP4<br></pre><pre>     MP1 ─────────────── shared<br>     MP2 ─────────────── shared<br>     MP3 ─────────────── shared<br>     MP4 ─────────────── clone only</pre><h3>The key Snowflake concept</h3><pre>BEFORE CHANGE<br>                 ─────────────<br>                ┌─────────────┐<br> SOURCE ───────►│ MP1 MP2 MP3 │<br>                └─────────────┘<br>                     ▲<br>                     │<br> CLONE ──────────────┘</pre><pre>             ZERO-COPY CLONE<br>             No data duplication<br></pre><pre>                 AFTER CHANGE<br>                 ────────────<br> SOURCE ───────► MP1 MP2 MP3</pre><pre> CLONE ────────► MP1 MP2 MP3<br>                    +<br>                  ► MP4<br>                    ↑<br>              new micro-partition</pre><p><strong>One subtle but important point:</strong> Snowflake doesn’t modify an existing micro-partition in place. Micro-partitions are immutable. When data changes, Snowflake writes new micro-partition(s), while the unchanged partitions can continue to be shared between the source and clone.</p><p>Let us test this real time in Snowflake to prove I am not lying!</p><h3>Snowflake Zero-Copy Cloning — Step-by-Step</h3><h3>Step 1: Create a Database</h3><p>Create a new database called CLONING_TEST.</p><pre>CREATE OR REPLACE DATABASE CLONING_TEST;</pre><h3>Step 2: Use the Database</h3><p>Set CLONING_TEST as the active database.</p><pre>USE DATABASE CLONING_TEST;</pre><h3>Step 3: Create the Customer Table</h3><p>Create a CUSTOMER table by copying the data from Snowflake&#39;s sample TPCH dataset.</p><pre>CREATE TABLE CLONING_TEST.PUBLIC.CUSTOMER AS<br>SELECT *<br>FROM SNOWFLAKE_SAMPLE_DATA.TPCH_SF100.CUSTOMER;</pre><h3>Step 4: Verify the Source Table</h3><p>Check the number of records in the CUSTOMER table.</p><pre>SELECT COUNT(*)<br>FROM CUSTOMER;</pre><p>This gives you the initial row count of the source table.</p><h3>Step 5: Create a Zero-Copy Clone</h3><p>Create CLONE_CUSTOMER as a clone of the CUSTOMER table.</p><pre>CREATE TABLE CLONE_CUSTOMER<br>CLONE CUSTOMER;</pre><p>At this point, Snowflake does <strong>not physically copy the existing micro-partitions</strong>. The source table and clone initially reference the same underlying micro-partitions.</p><h3>Step 6: Verify the Clone</h3><p>Check the row count of the cloned table.</p><pre>SELECT COUNT(*)<br>FROM CLONE_CUSTOMER;</pre><p>The row count should initially match the CUSTOMER table.</p><h3>Step 7: Modify the Clone</h3><p>Update the cloned table by changing the market segment from AUTOMOBILE to AUTO.</p><pre>UPDATE CLONE_CUSTOMER<br>SET C_MKTSEGMENT = &#39;AUTO&#39;<br>WHERE C_MKTSEGMENT = &#39;AUTOMOBILE&#39;;</pre><p>Because Snowflake micro-partitions are immutable, the existing shared micro-partitions are not modified directly. Snowflake creates new micro-partitions containing the changed data for the clone.</p><h3>Step 8: Check the Source Table</h3><p>Verify the distinct market segments in the original table.</p><pre>SELECT DISTINCT C_MKTSEGMENT<br>FROM CUSTOMER;</pre><p>The original CUSTOMER table should still contain:</p><pre>AUTOMOBILE</pre><p>This demonstrates that modifying the clone does <strong>not</strong> modify the source table.</p><h3>Step 9: Check the Clone</h3><p>Now check the distinct market segments in the cloned table.</p><pre>SELECT DISTINCT C_MKTSEGMENT<br>FROM CLONE_CUSTOMER;</pre><p>The clone should contain:</p><pre>AUTO</pre><p>instead of AUTOMOBILE for the rows that were updated.</p><h3>What This Demonstrates</h3><p>The experiment demonstrates <strong>Snowflake zero-copy cloning</strong>. Initially, CUSTOMER and CLONE_CUSTOMER share the same underlying micro-partitions, so no physical data is copied when the clone is created. Once the clone is modified, Snowflake creates new micro-partitions for the changed data while the original shared micro-partitions remain unchanged. Therefore, the source and clone become logically independent while avoiding an initial full copy of the data.</p><p>Whole code for test!</p><blockquote>create or replace database cloning_test;<br>use database cloning_test;</blockquote><blockquote>create table cloning_test.public.customer<br>as<br>select * from SNOWFLAKE_SAMPLE_DATA.TPCH_SF100.CUSTOMER</blockquote><blockquote>SELECT COUNT(*)<br>FROM CUSTOMER</blockquote><blockquote>CREATE TABLE CLONE_CUSTOMER CLONE CUSTOMER</blockquote><blockquote>SELECT COUNT(*)<br>FROM CLONE_CUSTOMER</blockquote><blockquote>UPDATE CLONE_CUSTOMER SET C_MKTSEGMENT = ‘AUTO’<br>WHERE C_MKTSEGMENT = ‘AUTOMOBILE’</blockquote><blockquote>SELECT DISTINCT C_MKTSEGMENT FROM CUSTOMER</blockquote><blockquote>SELECT DISTINCT C_MKTSEGMENT FROM CLONE_CUSTOMER</blockquote><p>Hope you enjoyed the article! Happy Coding Happy Learning!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=85618e4e1831" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Loop Engineering]]></title>
            <link>https://medium.com/@khasnobis.sanjit890/loop-engineering-714eb81dca64?source=rss-5bf4882d9b72------2</link>
            <guid isPermaLink="false">https://medium.com/p/714eb81dca64</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[loop-engineering]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Sanjit Khasnobis]]></dc:creator>
            <pubDate>Fri, 31 Jul 2026 22:12:29 GMT</pubDate>
            <atom:updated>2026-07-31T22:14:33.029Z</atom:updated>
            <content:encoded><![CDATA[<p>Todays topic is a story of Loop Engineering! Let us deep drive how Loop Engineering can change the future of development!</p><h3>Loop Engineering: The AI Skill That Will Replace Prompt Engineering</h3><p><em>“Stop prompting AI. Start building systems that prompt AI for you.”</em></p><p>For the past two years, everyone has been obsessed with <strong>Prompt Engineering</strong>.</p><p>Then came <strong>Context Engineering</strong>.</p><p>Soon after, <strong>Harness Engineering</strong> entered the conversation.</p><p>Now, in 2026, a new term is reshaping how developers and AI builders think about automation:</p><h3>Loop Engineering</h3><p>At first glance, it sounds like just another AI buzzword.</p><p>It’s not.</p><p>Loop Engineering represents one of the biggest mindset shifts since autonomous AI agents became mainstream.</p><p>Instead of asking:</p><blockquote><strong><em>“What should I tell the AI?”</em></strong></blockquote><p>You begin asking:</p><blockquote><strong><em>“How can I build a system that never needs me to ask again?”</em></strong></blockquote><p>And that’s a completely different game.</p><h3>From AI Assistant to AI Employee</h3><p>Imagine hiring a new employee.</p><p>Every five minutes you walk over and say:</p><ul><li>“Check our emails.”</li><li>“Review today’s bugs.”</li><li>“Update the report.”</li><li>“Create a pull request.”</li></ul><p>Eventually they ask:</p><p><em>“Why don’t you just give me a schedule?”</em></p><p>That’s exactly where AI is today.</p><p>Most people still work like this:</p><pre>Human<br> ↓<br>Prompt<br> ↓<br>AI<br> ↓<br>Result<br> ↓<br>Human writes another prompt</pre><p>Loop Engineering changes the architecture.</p><pre>Human<br> ↓<br>Designs the Loop<br> ↓<br>Loop decides next task<br> ↓<br>AI executes<br> ↓<br>Evaluator verifies<br> ↓<br>Loop repeats</pre><p>The human is no longer operating the machine.</p><p>The human builds the machine.</p><h3>The Evolution of AI Engineering</h3><p>Think of AI development as climbing four floors of a building.</p><h3>Floor 1 — Prompt Engineering</h3><p>You learn how to ask better questions.</p><p>Example:</p><p>Instead of</p><blockquote><em>“Write code”</em></blockquote><p>You write</p><blockquote><em>“Write a REST API in FastAPI with JWT authentication and unit tests.”</em></blockquote><p>Better prompt.</p><p>Better answer.</p><h3>Floor 2 — Context Engineering</h3><p>Now you realize prompts aren’t enough.</p><p>The AI also needs:</p><ul><li>documentation</li><li>previous conversations</li><li>company guidelines</li><li>APIs</li><li>project history</li></ul><p>Instead of improving the prompt…</p><p>…you improve <strong>what the AI can see</strong>.</p><h3>Floor 3 — Harness Engineering</h3><p>Now you’re designing the entire AI runtime.</p><p>You decide:</p><ul><li>Which tools it can access</li><li>Which APIs it can call</li><li>What counts as success</li><li>What permissions it receives</li></ul><p>You’re no longer writing prompts.</p><p>You’re configuring an AI worker.</p><h3>Floor 4 — Loop Engineering</h3><p>This is where everything changes.</p><p>The AI doesn’t wait for you anymore.</p><p>It wakes itself.</p><p>Finds work.</p><p>Does work.</p><p>Checks work.</p><p>Stores progress.</p><p>Repeats.</p><p>The human becomes the architect instead of the operator.</p><h3>What Exactly Is a Loop?</h3><p>Every successful AI loop follows five simple steps.</p><h3>1. Discover</h3><p>Find work automatically.</p><p>Example:</p><p>Every morning your AI checks:</p><ul><li>GitHub Issues</li><li>Failed CI pipelines</li><li>Customer support tickets</li><li>Slack messages</li></ul><p>Instead of waiting for instructions…</p><p>…it discovers work itself.</p><h3>2. Handoff</h3><p>Assign each task to an AI worker.</p><p>Think of it as a project manager.</p><p>One agent fixes bugs.</p><p>Another writes documentation.</p><p>Another updates tests.</p><p>Everyone works independently.</p><h3>3. Verify</h3><p>This might be the most important step.</p><p>Never let the same AI review its own work.</p><p>That’s like letting students grade their own exams.</p><p>Instead…</p><p>One AI writes.</p><p>Another AI reviews.</p><p>A third AI checks whether the code actually runs.</p><p>This dramatically improves reliability.</p><h3>4. Remember</h3><p>AI conversations disappear.</p><p>Memory shouldn’t.</p><p>Good loops save:</p><ul><li>task history</li><li>decisions</li><li>reports</li><li>project state</li><li>completed work</li></ul><p>The AI forgets.</p><p>Your repository shouldn’t.</p><h3>5. Repeat</h3><p>Finally…</p><p>The loop schedules itself.</p><p>Every hour.</p><p>Every morning.</p><p>Every deployment.</p><p>Every failed test.</p><p>No human required.</p><h3>A Real-World Example</h3><p>Imagine you’re managing an ecommerce website.</p><p>Without Loop Engineering, your day looks like this:</p><p>✔ Check analytics</p><p>✔ Review customer complaints</p><p>✔ Find broken pages</p><p>✔ Update inventory</p><p>✔ Generate reports</p><p>✔ Create Jira tickets</p><p>✔ Notify developers</p><p>Now imagine a loop.</p><p>Every morning at 7 AM it automatically:</p><ul><li>Reads Google Analytics</li><li>Detects traffic drops</li><li>Reviews customer reviews</li><li>Finds broken links</li><li>Opens GitHub issues</li><li>Creates Jira tasks</li><li>Sends Slack updates</li><li>Generates a daily report</li></ul><p>By the time you open your laptop…</p><p>Everything is already organized.</p><p>You didn’t manage the work.</p><p>You designed the system that manages the work.</p><h3>Why AI Should Never Grade Its Own Homework</h3><p>One fascinating discovery from the AI engineering community is this:</p><p>AI loves complimenting itself.</p><p>Ask an AI:</p><blockquote><em>“Is this code good?”</em></blockquote><p>The answer is almost always:</p><blockquote><em>“Yes.”</em></blockquote><p>Even when the code is terrible.</p><p>Why?</p><p>Because the AI remembers <strong>why</strong> it wrote that solution.</p><p>Humans do this too.</p><p>After writing an article, you read it five times and suddenly every sentence sounds brilliant.</p><p>Then someone else reads it…</p><p>…and immediately spots ten mistakes.</p><p>The same principle applies to AI.</p><p>Great Loop Engineering separates two roles:</p><p><strong>Generator</strong></p><p>Creates solutions.</p><p><strong>Evaluator</strong></p><p>Looks for problems.</p><p>Different responsibilities.</p><p>Different thinking.</p><p>Better outcomes.</p><h3>The Hidden Costs Nobody Talks About</h3><p>Loop Engineering is powerful.</p><p>But it’s not free.</p><h3>Verification Debt</h3><p>If your AI creates 100 pull requests overnight…</p><p>Someone still needs to verify them.</p><p>Automation saves time.</p><p>Blind trust creates risk.</p><h3>Knowledge Decay</h3><p>If AI writes everything…</p><p>Will you still understand your own system?</p><p>Many engineers already report opening files they no longer recognize.</p><p>Your project grows.</p><p>Your understanding shrinks.</p><h3>Automation Complacency</h3><p>When systems work perfectly…</p><p>People stop paying attention.</p><p>That’s dangerous.</p><p>The goal isn’t replacing judgment.</p><p>It’s replacing repetition.</p><h3>Token Explosion</h3><p>Loops can accidentally run forever.</p><p>Imagine a bug that retries endlessly.</p><p>Thousands of API calls.</p><p>Millions of tokens.</p><p>A very expensive morning.</p><p>Always add:</p><ul><li>spending limits</li><li>retry limits</li><li>timeout rules</li></ul><p>Automation should never have an unlimited credit card.</p><h3>The Biggest Mindset Shift</h3><p>Prompt Engineering asked:</p><blockquote><em>“How do I control AI?”</em></blockquote><p>Loop Engineering asks:</p><blockquote><em>“How do I build a system that controls AI?”</em></blockquote><p>That’s a huge difference.</p><p>One focuses on conversations.</p><p>The other focuses on infrastructure.</p><h3>Your First Loop Doesn’t Need to Be Fancy</h3><p>You don’t need an enterprise AI platform.</p><p>Start with something simple.</p><p>For example:</p><p>Every morning your AI could:</p><ol><li>Check failed tests.</li><li>Summarize GitHub issues.</li><li>Review yesterday’s commits.</li><li>Suggest fixes.</li><li>Send you one report.</li></ol><p>That’s already Loop Engineering.</p><p>As you grow, you can add:</p><ul><li>multiple AI agents</li><li>evaluators</li><li>automated testing</li><li>scheduling</li><li>persistent memory</li><li>integrations with Slack, Jira, GitHub, databases, and cloud services</li></ul><p>One loop becomes many.</p><p>Eventually your AI isn’t just answering questions.</p><p>It’s running parts of your workflow.</p><h3>Final Thoughts</h3><p>Loop Engineering isn’t about replacing developers.</p><p>It’s about replacing repetitive interactions.</p><p>The future isn’t humans writing thousands of prompts every day.</p><p>The future is humans designing intelligent systems that know <strong>when to think, what to do next, and when to ask for help</strong>.</p><p>The most valuable engineers won’t be the ones who write the most prompts.</p><p>They’ll be the ones who build the smartest loops.</p><p>Because in the age of autonomous AI, <strong>your competitive advantage isn’t how well you can talk to AI — it’s how well you can design systems that let AI work without you.</strong></p><p>Hope you enjoyed the article!</p><p>Happy Learning! Happy Coding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=714eb81dca64" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>