<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dipayan Dev on Medium]]></title>
        <description><![CDATA[Stories by Dipayan Dev on Medium]]></description>
        <link>https://medium.com/@dipayandev?source=rss-564f63b972e3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*bPmidwV2zt7zTo0Hmb_s7w.jpeg</url>
            <title>Stories by Dipayan Dev on Medium</title>
            <link>https://medium.com/@dipayandev?source=rss-564f63b972e3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 07:19:50 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dipayandev/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Build your AI-Agent in 5 minutes using Google ADK]]></title>
            <link>https://medium.com/@dipayandev/build-your-ai-agent-in-5-minutes-using-google-adk-5ac4074534dd?source=rss-564f63b972e3------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ac4074534dd</guid>
            <dc:creator><![CDATA[Dipayan Dev]]></dc:creator>
            <pubDate>Fri, 25 Jul 2025 15:05:43 GMT</pubDate>
            <atom:updated>2025-07-25T15:07:36.908Z</atom:updated>
            <content:encoded><![CDATA[<p>If you’ve been following the AI space lately, you’ve probably heard the growing buzz around <strong>AI agents</strong> — smart systems designed to make decisions and take actions on their own.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DbTBnnRAidzYa1MBXiVJyw.png" /></figure><p>Now, Google has stepped into this evolving space with its own powerful tool: the <a href="https://google.github.io/adk-docs/"><strong>Google Agent Development Kit (ADK)</strong></a>. This open-source Python library is built to make it easier for developers to design, test, and deploy intelligent AI agents. Whether you’re building a conversational chatbot or an automated backend process, ADK offers the structure and utilities you need to get started quickly and scale confidently.</p><p>In this article, I’ll walk you through a step-by-step process to build your own chatbot using <strong>three different AI agents</strong>.</p><p>Before we dive in, make sure to check out <a href="https://google.github.io/adk-docs/get-started/installation/">Google’s official installation and quickstart guide</a> to set up the environment and get ADK up and running.</p><p>In this demo, I’ll be using <strong>three different AI agents</strong>, each designed for a specific task:</p><ul><li><strong>Text-to-SQL Generator</strong> — Converts natural language queries into SQL statements.</li><li><strong>Multiplication Table Agent</strong> — Generates multiplication tables for any number you ask.</li><li><strong>Country Capital Agent</strong> — Provides the capital city for any given country.</li></ul><h4><strong>Root Agent (Mandatory step)</strong></h4><p>To build a multi-agent system using ADK, you need a designating a <strong>root agent</strong> (or orchestrator) that receives the initial user request.</p><pre>try:<br>    MODEL_GPT_4O = &quot;azure/gpt-4o&quot;<br>    root_agent = Agent(<br>        name=&quot;root_agent&quot;,<br>        model= MODEL_GPT_4O #here I am used GPT4-o, but you&#39;re free to chose any of the available LLM models<br>        description=&quot;Root agent of the chatbot&quot;,<br>        instruction=&quot;You&#39;re the root agent responsible for delegating your task to appropriate sub-agents as per user request.&quot;,<br>        sub_agents=[text_to_sql_agent, country_capital_agent, multiplication_table_agent],<br>    )<br>except Exception as e:<br>    print(f&quot;❌ Could not create or run GPT agent &#39;{MODEL_GPT_4O}&#39;. Check API Key and model name. Error: {e}&quot;)</pre><p>ADK makes <strong>automatic delegation</strong> simple by allowing you to define a list of sub_agents. When the <strong>main (or root) agent</strong> receives a user query, its underlying LLM doesn&#39;t just rely on it’s instructions — it also looks into the descriptions of all sub-agents.</p><p>If the system determines that a specific sub-agent (like a &quot;multiplication_table_agent&quot;) is better suited to handle the query, it will automatically delegate the task. It does this by triggering a special internal action transfer control that hands over control to the relevant sub-agent for that turn. The sub-agent then processes the request using its own unique model, tools, and prompt instructions.</p><p>I have added sub_agents parameter and pass list containing the text_to_sql_agent , country_capital_agent and multiplication_table_agent instances that we will create in next step.</p><pre>country_capital_agent = Agent(<br>    model=LiteLlm(model=MODEL_GPT_4O),<br>    name=&quot;country_capital_agent&quot;,<br>    instruction=&quot;You&#39;re an agent responsible for replying the capital of a country. If user ask anything else, you can say * I only know the capital of country*&quot;,<br>    description=&quot;You&#39;re an agent responsible for returning the capital of a country.&quot;,  # Crucial for delegation<br>    generate_content_config=types.GenerateContentConfig(temperature=0.01)<br>)<br>print(f&quot;✅ Agent &#39;{country_capital_agent.name}&#39; created using model &#39;{country_capital_agent.model}&#39;.&quot;)</pre><pre>text_to_sql_agent = Agent(<br>    model=LiteLlm(model=MODEL_GPT_4O),<br>    name=&quot;sql_agent&quot;,<br>    instruction=&quot;Convert text to SQL query&quot;,<br>    description=&quot;You&#39;re an agent responsible for convert user&#39;s natural language to SQL&quot;,  # Crucial for delegation<br>print(f&quot;✅ Agent &#39;{text_to_sql_agent.name}&#39; created using model &#39;{text_to_sql_agent.model}&#39;.&quot;)</pre><pre>multiplication_table_agent = Agent(<br>    model=LiteLlm(model=MODEL_GPT_4O),<br>    name=&quot;vector_multiplication_agent&quot;,<br>    instruction=&quot;When user asks to generate multiplication table, ask users **Please provide me the number**. If users provides the number, generate a multiplication table upto 10 and show me the result in tabular format&quot;,<br>    description=&quot;You&#39;re an agent dedicated for generating  multiplication table of an  3 dimensional vector and do multiplication&quot;,<br>)<br>print(f&quot;✅ Agent &#39;{multiplication_table_agent.name}&#39; created using model &#39;{multiplication_table_agent.model}&#39;.&quot;)</pre><p>Once you’ve saved your Python file, it’s time to test your chatbot. ADK makes this easy with its built-in <strong>DEV-UI</strong>. Just run the following command in your terminal:</p><pre>adk web</pre><p>This will launch a web-based interface where you can interact with your agent and test its responses.</p><p>In my example, I asked a simple question about a multiplication table. The <strong>root agent</strong> intelligently delegated the query to the multiplication_table_agent sub-agent — you can see this delegation in the visual graph on the UI. The response followed the exact format defined in that sub-agent’s instructions, showing how effectively ADK handles multi-agent coordination.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xjEos1aMP3nKJC2k6X5gGw.png" /></figure><p>Next, when I asked a question related to SQL queries, the root agent has again transferred the control to sql_agent</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JqxC-g2uCsNPPBE5wuTiRA.png" /></figure><p>This was a simple demo to help you get started with building your first chatbot using AI agents through Google’s ADK. The framework offers many more powerful features that make developing intelligent agents both seamless and scalable.</p><p>In future posts, I’ll dive deeper into advanced concepts, real-world applications, and best practices to unlock the full potential of ADK.</p><p><strong>If you found this helpful, give it a 👏, follow me for more hands-on AI tutorials, and drop a comment if there’s a specific topic you’d like me to cover next!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ac4074534dd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GCP Autoclass: Is it the Right Choice for Optimising Your Cloud Storage Costs]]></title>
            <link>https://medium.com/@dipayandev/gcp-autoclass-is-it-the-right-choice-for-optimising-your-cloud-storage-costs-a292347a2f58?source=rss-564f63b972e3------2</link>
            <guid isPermaLink="false">https://medium.com/p/a292347a2f58</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[google-cloud-storage]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Dipayan Dev]]></dc:creator>
            <pubDate>Sat, 15 Jul 2023 06:18:43 GMT</pubDate>
            <atom:updated>2023-07-15T12:16:44.315Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*0iqkX3YMK5fwJp6O.jpg" /></figure><p>In Google Cloud Next ’22, one of the most exciting announcements was the new <a href="https://cloud.google.com/storage/docs/autoclass"><strong>Autoclass</strong></a> feature for GCS Buckets.</p><p>GCP Autoclass, a feature offered by Google, introduces a management fee to simplify the process of object lifecycle management. This feature leverages the concept of Storage Class transition, where objects gradually cascade through different storage tiers over time, starting from Standard to Nearline, Coldline, and eventually Archive, which offers the most economical storage option.</p><h4>How Autoclass works</h4><p><strong>Autoclass</strong> is a <a href="https://cloud.google.com/java/docs/reference/google-cloud-storage/latest/com.google.storage.v2.Bucket.Autoclass">simple bucket-level setting</a> that is currently only available for <strong>newly-created </strong><a href="https://cloud.google.com/storage/docs/buckets"><strong>Cloud Storage buckets</strong></a>.</p><p>When <strong>Autoclass</strong> is enabled for a bucket, new objects are always written to the <a href="https://cloud.google.com/storage/docs/storage-classes#standard">Standard storage</a> class and Google automatically takes care of cooling and warming objects based on the access patterns.</p><ul><li>New objects are created in Standard storage.</li><li>If an object has not been accessed for 30 days, it automatically moves to Nearline storage.</li><li>If an object has not been accessed for 90 days, it automatically moves to Coldline storage.</li><li>If an object has not been accessed for 365 days, it automatically moves to Archive storage.</li></ul><p><strong>Objects less than 128KB are not transitioned as part of Autoclass</strong>. So, if you have way too many small files in your bucket, you won’t get the benefit of AutoClass management.</p><h4>Cost Analysis</h4><p>Cost analysis of 2 different use cases is done to compare Autoclass over non-Autoclass(consider fully unmanaged) enabled multi-regional bucket.</p><p><strong>Case I:</strong></p><p>A 481 TB multi-regional bucket where 300 TB accessed monthly and 181TB never accessed and only kept for compliance purposes. The bucket has 400M objects all greater than 128KB (all applicable for management fee)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8NDt8gsmDswy9j2ZtcSeGg.png" /></figure><p>The analysis shows how using Autoclass you can realise an annualised saving of ~40K.</p><p><strong>Case II:</strong></p><p>500 TB multi-regional bucket where every object is getting accessed at least once a month. The size of each object is 1MB. Total number of objects — 500M</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3ZtwML1YUQrJTfsH0b2Nvg.png" /></figure><p>In this scenario, we have a situation where no objects qualified for the cold storage class, and our bucket <strong>contained a huge number of small files.</strong> As a consequence, utilising Autoclass resulted <strong>in higher costs</strong> compared to an unmanaged bucket approach.</p><h4>Cons of Autoclass</h4><ul><li>Autoclass, despite its advantages, presents several noteworthy challenges that can impact the realisation of savings. One significant drawback is the extended timeframe required to achieve cost optimisation. In Autoclass, it takes 365 days for objects to transition to the Archive storage class, with no available means to accelerate this process. This means that even if you have objects within your bucket that are supposed to remain untouched for few years, Autoclass still <strong>needs to wait for a full year before they can be moved to Archive.</strong> <strong>Additionally, the storage class of these objects cannot be forcibly overwritten.</strong></li><li>Consider a scenario where you are aware that certain objects in your bucket are accessed only once every 300 days. If you were using a non-Autoclass bucket, you could manually overwrite the storage class to Archive and pay a retrieval fee only on the 300th day. This approach would result in significant cost savings compared to keeping the objects in the Standard class. <strong>However, with Autoclass, the objects will never be automatically transitioned to Archive and will remain in the Coldline storage tier.</strong></li><li>Another issue with Autoclass is once an object has been moved to Archive, even if you only need to query the data once in two years, the object will be bumped back to the Standard class with just a single read. As a consequence, the object again has to go through a 365-day waiting period before it can be moved back to the Archive class.</li><li>One notable limitation is that objects smaller than 128KB are not included in the Autoclass transition process. Consequently, if your bucket contains a substantial number of small files, you won’t reap the benefits of Autoclass management for those objects. This limitation poses a significant challenge, as small files can accumulate quickly and make up a substantial portion of your storage. Without the ability to leverage Autoclass for these files, you may miss out on potential cost savings and efficient storage utilisation.</li></ul><p>Generally, if you are not sure of the access pattern of the bucket, AutoClass could be a safer option. However, these limitations emphasise the importance of carefully considering the implications and trade-offs when opting for Autoclass as a cloud storage cost optimisation strategy.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a292347a2f58" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Everything you need to understand Data Partitioning in Spark]]></title>
            <link>https://medium.com/@dipayandev/everything-you-need-to-understand-data-partitioning-in-spark-487d4be63b9c?source=rss-564f63b972e3------2</link>
            <guid isPermaLink="false">https://medium.com/p/487d4be63b9c</guid>
            <category><![CDATA[parallelism]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[sharding]]></category>
            <category><![CDATA[partitioning]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Dipayan Dev]]></dc:creator>
            <pubDate>Sat, 30 Apr 2022 10:37:03 GMT</pubDate>
            <atom:updated>2022-04-30T10:50:26.381Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/731/1*6mr7EGQ-ZZ6n0m3Ie5YSAg.jpeg" /></figure><p><strong>What is Data Partitioning</strong></p><p>In a simple manner, partitioning in data engineering means splitting your data in smaller chunks based on a well defined criteria. In the context of Apache Spark, it can be defined as a dividing the dataset into multiple parts across the cluster.</p><p>The main idea behind data partitioning is to optimise your job performance. The job performance can be improved by carefully distributing the input dataset into smaller chunks and process those via Spark executors. But, as there is no single thumb rule on how to partition your data in a most optimised way, developers understanding of Spark internals and familiarity with the dataset become a crucial factor.</p><p><strong>Factor to consider while partitioning the data</strong></p><ul><li><strong>Avoid having too big or small files:</strong> Having bigger partitioned data will lead to some of the executor doing the heavy load work, while others are just sitting idle. We need to ensure that no executors in the cluster is sitting idle due to the skewed workload distribution across the executors. This will lead to increased data processing time because of weak utilisation of the cluster. <br>On the other hand, having too many small files may require lots of shuffling data on disk space, taking a lot of your network compute. <br>The recommendation is to keep your partition file size ranging 256MB to 1GB.</li></ul><p><strong>How to decide the partition key(s)?</strong></p><ul><li>Do not partition by columns having <strong>high cardinality</strong>. For example, don’t use your partition key such as roll_no, employee_ID etc. Instead your state code, country code, geo_code etc.</li><li>Partition data by specific columns that will be mostly used during filter and groupBy operations.</li></ul><p><strong>Playing with partitions</strong></p><p>Let’s do some experiments by using different partition methods and understand the partition number, file sizes, and folder structures generated using different Spark API methods. A small open-source COVID19 vaccation details dataset is taken from <a href="https://www.kaggle.com/datasets/gpreda/covid-world-vaccination-progress">Kaggle</a>.</p><pre>val DF = spark.read.option(“header”,true).csv(“/Users/dipayan/Desktop/country_vaccinations.csv”)<br>scala&gt; DF.count<br><strong>res7</strong>: <strong>Long</strong> = 86512</pre><p>We’ll use <strong>coalesce</strong>, <strong>repartition</strong> and <strong>partitionBy</strong> APIs of Spark and understand the difference between each of them.</p><p>The <em>coalesce</em> method <strong>reduces</strong> the <strong>number of partitions</strong> in a DataFrame. It <strong>avoids full shuffle</strong>, instead of creating new partitions, it shuffles the data using default Hash Partitioner , and <strong>adjusts into existing partitions</strong>, this means it can <strong>only decrease </strong>the number of partitions.</p><p>The <em>repartition</em> method can be used to either <strong>increase</strong> or <strong>decrease</strong> the <strong>number of partitions</strong> in a DataFrame. Repartition is a <strong>full Shuffle</strong> operation, where whole data is taken out from existing partitions and <strong>equally distributed</strong> into <strong>newly formed partitions</strong>.</p><p>In the below example, we will manually pass <strong>10</strong> as number of partitions, and see how the two methods distributes data differently.</p><pre>scala&gt; spark.time(DF.coalesce(10).write.mode(&quot;overwrite&quot;).option(&quot;header&quot;,true).csv(&quot;/Users/d0d02t4/dipayan/data/coalesce_test.csv&quot;))</pre><pre>Time taken: 416 ms</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6tQpYRsVOKFPBWKNAMvkrw.png" /></figure><pre>scala&gt; spark.time(DF.repartition(10).write.mode(“overwrite”).option(“header”,true).csv(“/Users/d0d02t4/Desktop/data/repartition_test.csv”))</pre><pre>Time taken: 639 ms</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e5DyBR-QkT-B7D8TCCqffQ.png" /></figure><p><strong>Observations</strong>:</p><ul><li>coalesce keeps five partitions even though we attempted to create 10.</li><li>Since<strong> </strong>coalesce avoids f<strong>ull shuffle</strong>, its more performant than repartition. See the execution time of <strong>416ms vs 639 ms</strong>.</li></ul><p><strong>partitionBy</strong>, instead of coalesce andrepartition functions, it will change the folder structure of the underlying data. In the above example, we can partition the data by <em>country </em>column, as it has low cardinality and more likely to be used as a filter condition.</p><pre>scala&gt; DF.select($&quot;country&quot;).distinct.count<br><strong>res6</strong>: <strong>Long</strong> = 223<br>scala&gt; DF.write.mode(“overwrite”).partitionBy(“country”).option(“header”,true).csv(“/Users/d0d02t4/Desktop/data/partition_by_test.csv”)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/1*Zx9U1aKUNSbp4HKb3k2Ddw.png" /></figure><p>We’ll see many folders got created equal to number of distinct countries. Data is split respectively based on specified column.</p><p>However, partitionBy has no control on how many part files will be created under each folder. In some use-case, you may want to create a single file per partitioned folders. In that case, you need to use repartition along with partitionBy as following:</p><pre>scala&gt; DF.repartition($”country”).write.mode(“overwrite”).partitionBy(“country”).option(“header”,true).csv(“/Users/d0d02t4/Desktop/data/partition_by_test.csv”)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h-VyZXawa4GTQs0SPfiW0w.png" /></figure><p>Hope that is able to explain all the different ways of data partitioning using Spark. Please drop a comment for recommendations and what all data engineering concept you want to understand.</p><p>Happy Learning!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=487d4be63b9c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>