<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Rammanchem on Medium]]></title>
        <description><![CDATA[Stories by Rammanchem on Medium]]></description>
        <link>https://medium.com/@rammanchem98?source=rss-86496f8dd53c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*HOPANFNXzT3yHSbG</url>
            <title>Stories by Rammanchem on Medium</title>
            <link>https://medium.com/@rammanchem98?source=rss-86496f8dd53c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 10:06:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rammanchem98/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[From Pipelines to Prompts: How I Discovered the Power of GenAI After Learning Traditional ML]]></title>
            <link>https://medium.com/@rammanchem98/from-pipelines-to-prompts-how-i-discovered-the-power-of-genai-after-learning-traditional-ml-cce30a2659fa?source=rss-86496f8dd53c------2</link>
            <guid isPermaLink="false">https://medium.com/p/cce30a2659fa</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[genai]]></category>
            <dc:creator><![CDATA[Rammanchem]]></dc:creator>
            <pubDate>Sat, 20 Jun 2026 09:20:12 GMT</pubDate>
            <atom:updated>2026-06-20T09:20:12.427Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hh-QLEP7u5wyCu05" /><figcaption>Photo by <a href="https://unsplash.com/@growtika?utm_source=medium&amp;utm_medium=referral">Growtika</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Just over a year ago, I began diving deep into data engineering and machine learning. Like many engineers starting this journey, I spent months learning the rigorous, standard workflows: data ingestion, data cleaning, strict schemas, and model training. I felt proud of the systems I built until I tried to solve the same problem using Generative AI.</p><p>That was the moment everything clicked.</p><p>If you are someone grinding through traditional machine learning pipelines, this article is for you. We are going to look at how I built an air quality prediction system using a traditional pipeline, the heavy infrastructure it required, and the exact moment I realized just how powerful Large Language Models (LLMs) and Vector Databases can be.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n0ifjNgxfDyCfkVuELm-CQ.png" /><figcaption>Figure 1: Architectural comparison between a traditional ML pipeline and a GenAI Agent pattern.</figcaption></figure><h3><strong>1. The Grind: Building the Traditional ML Pipeline</strong></h3><p>To predict air quality (categorizing it as low, medium, high, or hazardous), my original project relied on a highly structured machine learning pipeline. I used an external API to fetch weather and pollution metrics, ingested that raw data, and eventually fed it into a multinomial Logistic Regression model.</p><p>The engineering overhead was massive:</p><ul><li><strong>Strict Storage and Contracts:</strong> I wrote <strong>ingest_data.py</strong> to pull live data, format it into clean JSON, and push it into Google BigQuery with rigid schema definitions. If the schema changed by one column, the pipeline failed.</li><li><strong>Manual Preprocessing:</strong> In preprocessing.py, I had to explicitly isolate exactly 10 numerical features (like temperature, pressure, humidity, and wind speed) and fill any missing data safely using median imputation.</li><li><strong>The 3-Way Split:</strong> Inside my automated orchestration framework (training_pipeline.json), I had to build a custom data-split component. This split the data into three separate buckets validation, train, and test while carefully scaling features to avoid data leakage.</li></ul><pre>import pandas as pd<br>from google.cloud import bigquery<br>from sklearn.model_selection import train_test_split<br>from sklearn.preprocessing import StandardScaler<br>from sklearn.linear_model import LogisticRegression<br><br># Load raw structured data directly from BigQuery<br>bq_client = bigquery.Client(project=&quot;london-air-quality-pipeline&quot;)<br>query = &quot;SELECT * FROM `world_weather_dataset.world_air_quality`&quot;<br>df = bq_client.query(query).to_dataframe()<br><br># Explicit Feature Engineering: Isolate the strict 10 feature matrix<br>features = [&quot;temp&quot;, &quot;pressure&quot;, &quot;humidity&quot;, &quot;wind_speed&quot;, &quot;wind_direction&quot;, &quot;...&quot;]<br># Median imputation<br>df_features = df[features].fillna(df[features].median()) # Median imputation required<br><br># Target Calculation<br>df[&#39;target&#39;] = df[&#39;pm25_value&#39;].apply(lambda x: &#39;high&#39; if x &gt; 55.4 else &#39;low&#39;)<br><br># 2. Strict 3-Way Data Split to prevent leakage<br># Isolate test set<br>x_temp, x_test, y_temp, y_test = train_test_split(df_features, df[&#39;target&#39;], test_size=0.2)<br><br># Isolate train and validation sets<br>x_train, x_val, y_train, y_val = train_test_split(x_temp, y_temp, test_size=0.25)<br><br># 3. Explicit Scaling<br>scaler = StandardScaler()<br>x_train_scaled = scaler.fit_transform(x_train)<br>x_val_scaled = scaler.transform(x_val)<br><br># 4. Model Training<br>model = LogisticRegression(multi_class=&quot;multinomial&quot;, solver=&quot;lbfgs&quot;)<br>model.fit(x_train_scaled, y_train)<br><br># Output deterministic predictions<br>predictions = model.predict(x_val_scaled)</pre><pre>Model Prediction Output:<br>[&#39;high&#39;]</pre><h3><strong>2. The “Damn” Moment: Bypassing the Pipeline</strong></h3><p>After spending months perfecting that architecture, I decided to try a different approach for a newer project. I signed up for the Gemini SDK, initialized a foundational model, and did something that felt completely wrong as an ML engineer:<strong> </strong>I threw the raw, messy OpenWeatherMap API JSON payload directly into the prompt.</p><p>No preprocessing.py to impute medians. No StandardScaler to normalize features. No LogisticRegression model to train.</p><p>I simply gave the LLM a system prompt defining the air quality thresholds and asked it to analyse the payload.</p><pre>from google import genai<br><br># 1. Initialize the GenAI client<br>client = genai.Client()<br><br># 2. Simulate the raw, unformatted JSON payload straight from the API<br>raw_json_payload = &quot;&quot;&quot;<br>{<br>  &quot;city&quot;: &quot;London&quot;,<br>  &quot;main&quot;: {&quot;temp&quot;: 15.5, &quot;pressure&quot;: 1012, &quot;humidity&quot;: 76},<br>  &quot;wind&quot;: {&quot;speed&quot;: 4.1, &quot;deg&quot;: 180},<br>  &quot;pollution&quot;: {&quot;pm2_5&quot;: 42.1}<br>}<br>&quot;&quot;&quot;<br><br># 3. Construct the zero-shot prompt<br># Define the threshold logic directly in natural language (e.g., &lt;= 12.0 is low)<br>prompt = f&quot;&quot;&quot;<br>You are an expert environmental data classifier. Your task is to analyze raw weather and pollution API payloads and determine the air quality category based strictly on the PM2.5 value.<br><br>Use the following strict PM2.5 thresholds:<br>- 0.0 to 12.0 = low<br>- 12.1 to 35.4 = medium<br>- 35.5 to 55.4 = high<br>- 55.5 and above = hazardous<br><br>Analyze the provided JSON payload, extract the PM2.5 value, and return ONLY the single classification word (low, medium, high, or hazardous). Do not include any other text, reasoning, punctuation, or formatting.<br><br>Raw Data to Analyze: <br>{raw_json_payload}<br>&quot;&quot;&quot;<br><br># 4. Execute the generation (bypassing the traditional ML pipeline entirely)<br>response = client.models.generate_content(<br>    model=&quot;gemini-3.5-flash&quot;,<br>    contents=prompt<br>)<br><br>print(f&quot;LLM Classification: {response.text}&quot;)<br><br>&gt; LLM Classification: Hazardous </pre><p>When the model read the raw text block, understood the relationship between the variables, and instantly returned the correct air quality classification, my jaw dropped. <em>“Damn,”</em> I thought. <em>I just spent weeks building an infrastructure pipeline to calculate something this model can infer in three seconds of prompt engineering.</em></p><p>The LLM didn’t need numbers in a neat matrix, it possessed semantic reasoning capability out of the box.</p><h3>3. Scaling Up: Handling the Unstructured with RAG</h3><p>The zero-shot prompt was incredible, but real-world engineering requires scale and context. In traditional ML, if I wanted my model to account for a local news alert about regional wildfires or a text-based smog warning from the municipality, I couldn’t do it because <strong>Traditional models require rigid numerical arrays, meaning I would have to build an entirely separate NLP pipeline just to convert that text into a usable number.</strong></p><p>With GenAI, I swapped out the rigid BigQuery schemas for a vector database: <strong>ChromaDB</strong>.</p><p>Instead of parsing columns, the architecture became a Retrieval-Augmented Generation (RAG) pattern:</p><ol><li><strong>Vector Storage:</strong> I fed raw weather data, text alerts, and environmental news into ChromaDB as embeddings.</li><li><strong>Semantic Retrieval:</strong> When querying for a specific city’s air quality, ChromaDB pulled the most relevant context even if that context was a paragraph of text instead of a floating-point number.</li><li><strong>Context-Informed Generation:</strong> The Gemini SDK consumed this retrieved context, combined it with the user query, and generated a highly accurate, reasoned prediction.</li></ol><pre>import chromadb<br>from google import genai<br>from google.genai import types<br><br># 1. Initialize Clients <br>client = genai.Client()<br>client.automatic_function_calling = False <br><br>chroma_client = chromadb.PersistentClient(path=&quot;./chroma_db&quot;)<br>collection = chroma_client.get_or_create_collection(name=&quot;city_air_quality&quot;)<br><br># 2. Define the Retrieval Tool<br>def search_air_quality_db(query: str) -&gt; str:<br>    &quot;&quot;&quot;Searches the local vector database for real-time air quality metrics.&quot;&quot;&quot;<br>    print(f&quot;[TOOL] Executing semantic search for: &#39;{query}&#39;&quot;)<br>    embed_res = client.models.embed_content(model=&#39;gemini-embedding-2&#39;, contents=query)<br>    results = collection.query(query_embeddings=[embed_res.embeddings[0].values], n_results=3)<br>    return &quot;\n&quot;.join(results[&quot;documents&quot;][0])<br><br># 3. Execute the Manual RAG Agent Loop<br>def run_agent(user_prompt: str):<br>    config = types.GenerateContentConfig(<br>        tools=[search_air_quality_db],<br>        system_instruction=&quot;You are an air quality assistant. Use your search tool to retrieve data.&quot;<br>    )<br><br>    # Turn 1: Ask Gemini how to proceed. It will return a function call request instead of text.<br>    response = client.models.generate_content(<br>        model=&quot;gemini-3.5-flash&quot;,<br>        contents=user_prompt,<br>        config=config<br>    )<br><br>    # Check if the LLM requested our vector search tool<br>    if response.function_calls:<br>        tool_responses = []<br>        for function_call in response.function_calls:<br>            # Extract the query the LLM generated and run our local python function<br>            tool_output = search_air_quality_db(query=function_call.args[&quot;query&quot;])<br>            <br>            # Package the raw database text to send back to the LLM<br>            response_part = types.Part.from_function_response(<br>                name=function_call.name,<br>                response={&quot;result&quot;: tool_output}<br>            )<br>            tool_responses.append(response_part)<br>            <br>        # Turn 2: Send the original prompt + the LLM&#39;s request + our database context<br>        final_response = client.models.generate_content(<br>            model=&quot;gemini-3.5-flash&quot;,<br>            contents=[<br>                user_prompt,<br>                response.candidates[0].content,<br>                types.Content(role=&quot;tool&quot;, parts=tool_responses)<br>            ],<br>            config=config<br>        )<br>        print(f&quot;\nFinal Manual Agent Response:\n{final_response.text}&quot;)<br><br>if __name__ == &quot;__main__&quot;:<br>    run_agent(&quot;Compare the current air metrics of New Delhi and Johannesburg.&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dyMfmjnfh72o1_KmS7kR6Q.png" /><figcaption><em>Figure 2 : Execution logs of the manual RAG loop capturing the background agent-to-tool orchestration turn.</em></figcaption></figure><h3><strong>4. Engineering Reality: The Trade-offs</strong></h3><p>Seeing the power of LLMs made me realize how fast engineering is changing, but it also taught me that every architecture has trade-offs. As I grow in my engineering career, I’m learning that GenAI isn’t a magic replacement for traditional ML it’s a different tool for a different job.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*QEXr4XMS0n5ER4STeTiYdw.png" /><figcaption><em>Figure 3 : Paradigm comparison: Traditional ML Pipelines vs GenAI Agent Patterns.</em></figcaption></figure><h3><strong>Final Thoughts</strong></h3><p>Stepping out of the rigid boundaries of traditional machine learning pipelines opened my eyes to what software engineering looks like today. Building the traditional pipeline wasn’t a waste of time it taught me data discipline, splitting strategies, and orchestration.</p><p>But discovering the reasoning power of LLMs changed how I think about problem-solving. If you haven’t taken your raw data and passed it to a modern LLM yet, try it. You might just have your own “damn” moment.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cce30a2659fa" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>