<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Chethan Prabhas on Medium]]></title>
        <description><![CDATA[Stories by Chethan Prabhas on Medium]]></description>
        <link>https://medium.com/@jc.prabhaswork?source=rss-8778549674bb------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*D33VFusSRtykaOhfNyxtcg.jpeg</url>
            <title>Stories by Chethan Prabhas on Medium</title>
            <link>https://medium.com/@jc.prabhaswork?source=rss-8778549674bb------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 23:02:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jc.prabhaswork/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[From Data Engineering to Agentic AI: Building an Intelligent HR Assistant with Databricks]]></title>
            <link>https://medium.com/@jc.prabhaswork/from-data-engineering-to-agentic-ai-building-an-intelligent-hr-assistant-with-databricks-b7494076acc3?source=rss-8778549674bb------2</link>
            <guid isPermaLink="false">https://medium.com/p/b7494076acc3</guid>
            <category><![CDATA[langgraph]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[generative-ai-use-cases]]></category>
            <dc:creator><![CDATA[Chethan Prabhas]]></dc:creator>
            <pubDate>Mon, 09 Nov 2026 15:56:48 GMT</pubDate>
            <atom:updated>2026-11-09T15:56:48.136Z</atom:updated>
            <content:encoded><![CDATA[<h3>From RAG to Agentic AI: Building an Intelligent HR Assistant with LangGraph and Databricks — Part 2</h3><p>In <a href="https://medium.com/@jc.prabhaswork/from-data-engineering-to-agentic-ai-building-an-intelligent-hr-assistant-with-databricks-f2cf8ecdcb0d">Part 1</a>, we built the knowledge foundation of our PeopleOps HR Assistant.</p><p>We processed HR policy PDFs, generated embeddings using Databricks GTE Large EN, stored them in Delta Lake, and configured Databricks Vector Search to retrieve relevant policy information.</p><p>But retrieving information is only one part of an enterprise assistant.</p><p>Consider these employee requests:</p><ul><li>“What is the company’s leave policy?”</li><li>“Show me my leave balance.”</li><li>“Can you check my employee details?”</li><li>“I have an attendance issue. Can you raise an HR ticket?”</li></ul><p>These questions require different actions.</p><p>The first needs document retrieval. The second and third need structured database queries. The fourth requires an actual write operation.</p><p>We need something that can understand the request, select the appropriate capability, execute it, and return a useful response.</p><p>That’s where Agentic AI becomes interesting.</p><p>In this article, I’ll explain how I connected the RAG pipeline with SQL tools and built a conversational HR assistant using LangGraph, Databricks GPT OSS 120B, Streamlit, and MLflow.</p><p><strong>GitHub Repository:</strong> <a href="https://github.com/Prabhas92/PeopleOps_HR_Assistant">https://github.com/Prabhas92/PeopleOps_HR_Assistant</a></p><p><strong>Youtube Demo: </strong><a href="https://youtu.be/EdR3X5RBDJo?si=ZCcKzHSzHJteWVxq">https://youtu.be/EdR3X5RBDJo?si=ZCcKzHSzHJteWVxq</a></p><h3>1. Why do we need an AI agent?</h3><p>A basic RAG application generally follows a predefined retrieval workflow:</p><p>Question → Retrieve Documents → Generate Answer</p><p>But an enterprise assistant needs to handle multiple types of requests.</p><p>For example:</p><p>Employee questionRequired actionWhat is the leave policy?Search HR documentsWhat is my leave balance?Query leave tableShow my employee detailsQuery employee tableRaise an attendance issuePrepare and create an HR ticket</p><p>We could write separate conditions for every possible question.</p><p>However, this becomes difficult to maintain as the application grows.</p><p>Instead, I used an AI agent that can select from a set of predefined tools.</p><p>The LLM understands the employee’s intent, while the tools perform the actual operations.</p><p>This is the key idea behind the agent architecture.</p><h3>2. Understanding the architecture</h3><p>The application consists of four major components:</p><ol><li><strong>LLM:</strong> Understands employee questions and decides which tool to use.</li><li><strong>Tools:</strong> Perform specific operations such as document retrieval and SQL queries.</li><li><strong>Orchestration:</strong> Manages the agent’s tool selection and execution.</li><li><strong>Frontend:</strong> Provides the conversational interface.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VebRXYZ5pfnDjqwvrYswGA.png" /></figure><p>I used:</p><ul><li>Databricks GPT OSS 120B as the LLM.</li><li>LangGraph and LangChain for agent orchestration.</li><li>Databricks Vector Search for policy retrieval.</li><li>Databricks SQL Warehouse for structured data access.</li><li>Streamlit for the frontend.</li></ul><p>The high-level flow:</p><p>Employee → Streamlit → LangGraph Agent → LLM → Selected Tool → Data Source → Agent Response → Streamlit</p><h3>3. Step 1 — Connect the LLM</h3><p>The LLM is responsible for understanding natural language and deciding which tool is appropriate.</p><p>I used the Databricks GPT OSS 120B model endpoint.</p><p>Why an LLM?</p><p>Employees should not need to understand database table names, SQL syntax, vector indexes, or application logic.</p><p>They should simply be able to ask questions naturally.</p><p>For example:</p><p>“Can you check how many earned leaves I have remaining?”</p><p>The agent should understand that this is a leave-balance request and select the corresponding SQL tool.</p><p>The LLM does not need to invent the employee’s leave balance. It should obtain the value from the actual data source through the tool.</p><h3>4. Step 2 — Build specialized tools</h3><p>Instead of giving the agent unrestricted database access, I created specialized tools for specific business operations.</p><p>The project includes four main tools:</p><p>ToolResponsibilityData sourceemployee_details_toolRetrieves employee detailsemployeesleave_balance_toolRetrieves leave entitlements and balancesleave_balancehr_policy_toolSearches HR policy documentsVector Searchcreate_hr_ticketCreates HR support ticketshr_support_tickets</p><p>This design keeps responsibilities separated.</p><p>For example, the leave-balance tool should only handle leave-related information. It should not be responsible for searching policy documents or creating tickets.</p><h3>Why use tools instead of allowing the LLM to generate everything?</h3><p>Because the LLM is good at understanding language, but business data should come from reliable sources.</p><p>For example:</p><p>If an employee asks for their remaining leave balance, the LLM should not calculate or guess it from general knowledge.</p><p>It should call the SQL tool, retrieve the actual record, and explain the result.</p><p>This separation between reasoning and execution is one of the most important design decisions in an enterprise AI application.</p><h3>5. Step 3 — Integrate Databricks SQL</h3><p>Employee details and leave balances are structured data.</p><p>I used Databricks SQL Warehouse to access the corresponding Delta tables.</p><p>The application uses SQL queries through the Databricks SQL Connector.</p><p>A simplified example of the approach:</p><pre>def run_query(query, parameters=None):<br>    with connection.cursor() as cursor:<br>        cursor.execute(query, parameters)<br>        return cursor.fetchall()</pre><p>The actual application uses parameterized SQL for data access.</p><p>Why is that important?</p><ul><li>It helps avoid unsafe SQL string construction.</li><li>It separates query structure from user-supplied values.</li><li>It makes database operations easier to manage.</li></ul><p>Another important consideration is employee identity.</p><p>A production application should authenticate the employee and enforce authorization on the server side before returning employee-specific information.</p><p>A prompt instruction alone is not an access-control mechanism.</p><h3>6. Step 4 — Connect RAG as an agent tool</h3><p>Remember the Vector Search pipeline from Part 1?</p><p>Now we expose that retrieval capability to the agent through hr_policy_tool.</p><p>When an employee asks:</p><p>“What is the company’s hybrid work policy?”</p><p>The agent selects the policy retrieval tool.</p><p>The tool searches the Databricks Vector Search index and returns relevant policy chunks.</p><p>Those retrieved chunks are then used as context to generate the response.</p><p>This is where the two layers come together:</p><ul><li>RAG handles knowledge retrieval.</li><li>The agent decides when to use RAG.</li><li>The LLM generates the conversational response.</li></ul><p>The agent can now work with both unstructured policy documents and structured employee data.</p><h3>7. Step 5 — Implement human-in-the-loop ticket creation</h3><p>This was one of the most interesting workflows in the project.</p><p>Consider an employee saying:</p><p>“I have an issue with my attendance records. Can you raise an HR ticket?”</p><p>Should the agent immediately insert a record into the database?</p><p>I decided that it should not.</p><p>Creating a support ticket is a write operation. The employee should have an opportunity to review and confirm the proposed details before the application executes it.</p><p>My workflow:</p><ol><li>Employee describes the issue.</li><li>Agent understands the request.</li><li>Agent determines the ticket category, subject, description, and priority.</li><li>Agent presents a summary of the proposed ticket.</li><li>Employee confirms whether to create it.</li><li>Only after explicit confirmation does the ticket tool execute.</li><li>The assistant returns the ticket ID and status after successful creation.</li></ol><p>For example:</p><p><strong>Employee:</strong></p><p>“I have an issue with my attendance records. Can you raise an HR ticket?”</p><p><strong>Assistant:</strong></p><p>“I can create an HR support ticket for your attendance issue. Here are the proposed details…”</p><p><strong>Employee:</strong></p><p>“Yes, create it.”</p><p><strong>Assistant:</strong></p><p>“Your ticket has been created successfully. Here is your ticket ID…”</p><p>If the employee says no, the ticket should not be created.</p><h3>Why does this matter?</h3><p>This introduces a human-in-the-loop step into the agent workflow.</p><p>The LLM can understand and prepare an action, but the actual database write happens only after the employee confirms.</p><p>This is an important design pattern for enterprise AI applications where actions have consequences.</p><h3>8. Step 6 — Orchestrate everything using LangGraph</h3><p>Now we have an LLM and four specialized tools.</p><p>We need an orchestration layer to connect them.</p><p>I used LangGraph and LangChain to build the agent.</p><p>At a high level, the agent:</p><ol><li>Receives the employee’s message.</li><li>Interprets the intent.</li><li>Selects an appropriate tool when needed.</li><li>Executes the tool.</li><li>Receives the tool result.</li><li>Generates the final response.</li></ol><p>The key benefit is that the employee doesn’t need to select the underlying operation manually.</p><p>The agent decides whether the request requires policy retrieval, employee data, leave information, or ticket creation.</p><p>This creates a more natural conversational experience.</p><h3>9. Step 7 — Build the Streamlit application</h3><p>An AI backend is useful, but employees need a simple way to interact with it.</p><p>I used Streamlit to create the conversational frontend.</p><p>The application provides:</p><ul><li>A chat interface.</li><li>Employee questions and assistant responses.</li><li>Integration with the PeopleOps agent.</li><li>A simple way to test different HR workflows.</li></ul><p>The frontend calls the backend function, which submits the question to the LangGraph agent and returns its final response.</p><p>This separation between frontend and backend makes the application easier to maintain.</p><p>The repository keeps the application code under the app/ directory:</p><pre>app/<br>├── app.py<br>├── app.yaml<br>└── peopleops_agent.py</pre><h3>10. Step 8 — Deploy using Databricks Apps</h3><p>After building and testing the application, I deployed it using Databricks Apps.</p><p>The application requires access to the relevant Databricks resources, including:</p><ul><li>SQL Warehouse</li><li>GPT OSS 120B model endpoint</li><li>GTE Large EN embedding endpoint</li><li>Vector Search index</li><li>Required Unity Catalog resources and permissions</li></ul><p>One important deployment lesson is that local development access and application service-principal access are not necessarily the same.</p><p>The deployed application needs its own appropriate resource permissions.</p><p>Credentials should be managed securely, and access should follow the principle of least privilege.</p><h3>11. Step 9 — Add observability with MLflow</h3><p>An AI agent can produce a response, but how do we understand what happened behind the scenes?</p><p>For example:</p><ul><li>Did the agent select the correct tool?</li><li>Did Vector Search retrieve relevant policy chunks?</li><li>Did the SQL tool execute successfully?</li><li>Did the final answer use the retrieved information correctly?</li></ul><p>This is where observability becomes important.</p><p>I incorporated MLflow tracing and evaluation into the project to inspect agent behaviour and assess response quality.</p><p>Some useful evaluation scenarios include:</p><p>Test scenarioWhat to validateHR policy questionRetrieval relevance and answer groundingEmployee detailsCorrect tool selection and data retrievalLeave balanceAccurate structured data responseTicket creationConfirmation before database insertionUnrelated questionAppropriate handling of unsupported requests</p><p>A working demo is not automatically a reliable AI application.</p><p>Testing retrieval quality, tool selection, and final responses is an important part of the development process.</p><h3>12. What did we build?</h3><p>We now have an end-to-end conversational PeopleOps assistant.</p><p>An employee can:</p><ul><li>Ask questions about HR policies.</li><li>Retrieve employee information.</li><li>Check leave balances.</li><li>Raise HR support tickets with explicit confirmation.</li></ul><p>The application brings together:</p><ul><li>Document processing</li><li>Embedding</li><li>RAG</li><li>Databricks Vector Search</li><li>Delta Lake</li><li>SQL integration</li><li>LLMs</li><li>LangGraph agent orchestration</li><li>Streamlit</li><li>Databricks Apps</li><li>MLflow tracing and evaluation</li></ul><h3>13. Key engineering lessons</h3><p>Building this project helped me understand that Agentic AI is not simply about connecting an LLM to multiple tools.</p><p>Some important lessons:</p><p><strong>1. Keep reasoning separate from execution.</strong></p><p>The LLM should understand intent and select tools. Reliable systems should retrieve business information from the appropriate data sources.</p><p><strong>2. Design tools around business responsibilities.</strong></p><p>Small, specialized tools are easier to test, maintain, and control.</p><p><strong>3. Treat write operations carefully.</strong></p><p>Human confirmation is valuable when an agent is about to perform an action that changes business data.</p><p><strong>4. Security must be enforced outside the prompt.</strong></p><p>Identity verification, authorization, and database permissions need to be implemented at the application and data layers.</p><p><strong>5. Evaluate the complete workflow.</strong></p><p>A good response depends on retrieval quality, tool selection, execution, and generation — not just the LLM.</p><h3>14. What’s next?</h3><p>This project is a practical foundation, not a claim of production-ready enterprise HR automation.</p><p>Some improvements I would explore next:</p><ul><li>Identity-aware employee authentication and authorization.</li><li>Better conversation memory and context management.</li><li>HR ticket tracking and status updates.</li><li>Improved document version management.</li><li>Retrieval reranking and evaluation.</li><li>Automated policy ingestion</li><li>More comprehensive agent evaluation.</li></ul><h3>Final thoughts</h3><p>This project helped me connect Data Engineering with Generative AI in a practical way.</p><p>Rather than building another standalone chatbot, I wanted to understand how an AI application can combine structured data, unstructured documents, retrieval, tool execution, and human confirmation into one workflow.</p><p>The most valuable part was seeing how these individual components work together to solve a real business-style problem.</p><p>If you’re learning Databricks, LangGraph, RAG, or Agentic AI, I hope this implementation gives you a useful starting point.</p><p><strong>Explore the complete source code and project documentation:</strong></p><p><a href="https://github.com/Prabhas92/PeopleOps_HR_Assistant">https://github.com/Prabhas92/PeopleOps_HR_Assistant</a></p><p>If you build something similar, I’d love to hear what you learn and what you would improve.</p><p>#Databricks #LangGraph #AgenticAI #GenerativeAI #RAG #Python #SQL #MLflow #DataEngineering #AIEngineering</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b7494076acc3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Data Engineering to Agentic AI: Building an Intelligent HR Assistant with Databricks]]></title>
            <link>https://medium.com/@jc.prabhaswork/from-data-engineering-to-agentic-ai-building-an-intelligent-hr-assistant-with-databricks-f2cf8ecdcb0d?source=rss-8778549674bb------2</link>
            <guid isPermaLink="false">https://medium.com/p/f2cf8ecdcb0d</guid>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[mosaic-ai-vector-search]]></category>
            <category><![CDATA[delta-lake]]></category>
            <category><![CDATA[agentic-rag]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Chethan Prabhas]]></dc:creator>
            <pubDate>Mon, 09 Nov 2026 15:46:57 GMT</pubDate>
            <atom:updated>2026-11-09T15:57:23.088Z</atom:updated>
            <content:encoded><![CDATA[<h3>Beyond PDFs: Building an Enterprise HR Knowledge Assistant with Databricks Vector Search and RAG — Part 1</h3><h3>From scattered HR documents to a searchable knowledge base</h3><p>Imagine joining a company and having a simple question:</p><ul><li>How many earned leave days can I carry forward?</li><li>What is the company’s hybrid work policy?</li><li>Are there any restrictions on travel reimbursements?</li><li>What is the process for reporting a workplace concern?</li></ul><p>You ask the HR team. They search through documents, check the latest policy, and eventually get back to you.</p><p>Now imagine hundreds of employees asking similar questions every day.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hi-hn7DmPkPpLz41M7cmXw.png" /></figure><p>The problem isn’t necessarily that the information doesn’t exist. It’s that the information is scattered across documents, and finding the correct answer takes time.</p><p>This inspired me to build <strong>PeopleOps HR Assistant</strong>, an end-to-end Generative AI application using Databricks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tDRYqho49UK9rDxS_aAucQ.png" /></figure><p>In this first article, I’ll walk through the foundation of the solution: transforming HR policy documents into a searchable knowledge base using document processing, embeddings, Delta Lake, and Databricks Vector Search.</p><p><strong>GitHub Repository:</strong> <a href="https://github.com/Prabhas92/PeopleOps_HR_Assistant">https://github.com/Prabhas92/PeopleOps_HR_Assistant</a></p><p><strong>Youtube Demo: </strong><a href="https://youtu.be/EdR3X5RBDJo?si=ZCcKzHSzHJteWVxq">https://youtu.be/EdR3X5RBDJo?si=ZCcKzHSzHJteWVxq</a></p><h3>1. Understanding the problem before choosing the technology</h3><p>Before writing code, I broke the problem into three parts:</p><ol><li>HR policies are stored in unstructured documents such as PDFs.</li><li>Employee information and leave balances are stored in structured tables.</li><li>Employees need a simple conversational interface to access both.</li></ol><p>For this article, we’ll focus on the first problem: making unstructured HR documents searchable and useful to an LLM.</p><p>A traditional keyword search can find exact words, but it may struggle when employees ask questions using different wording.</p><p>For example:</p><p><strong>Policy wording:</strong></p><p>“Employees are eligible to carry forward a maximum of two earned leave days.”</p><p><strong>Employee question:</strong></p><p>“Can I transfer my unused annual leave to next year?”</p><p>The wording is different, but the meaning is similar.</p><p>This is where semantic search and Retrieval-Augmented Generation (RAG) become useful.</p><h3>2. Understanding the RAG architecture</h3><p>RAG combines two capabilities:</p><ul><li><strong>Retrieval:</strong> Find relevant information from an external knowledge source.</li><li><strong>Generation:</strong> Use an LLM to generate an answer based on the retrieved information.</li></ul><p>Instead of asking an LLM to answer from its training knowledge, we retrieve relevant company policy content and provide it as context.</p><p>The high-level workflow looks like this:</p><p>HR Policy PDFs → Text Extraction → Text Cleaning → Chunking → Embeddings → Delta Table → Vector Search → Relevant Context → LLM Answer</p><p>This approach helps ground responses in company-specific information.</p><h3>3. Step 1 — Collect and process HR documents</h3><p>I started with sample HR documents covering:</p><ul><li>Leave policy</li><li>Code of conduct</li><li>Holiday calendar</li><li>Hybrid work policy</li><li>Travel and expense policy</li></ul><p>The documents are stored in the project’s hr-docs/ directory.</p><p>The first challenge is extracting useful text from PDFs.</p><p>For this, I used Python PDF-processing libraries to extract the document content.</p><p>A simplified example:</p><pre>from PyPDF2 import PdfReader</pre><pre>def extract_text(pdf_path):<br>    reader = PdfReader(pdf_path)</pre><pre>    pages = []</pre><pre>    for page in reader.pages:<br>        text = page.extract_text()</pre><pre>        if text:<br>            pages.append(text)</pre><pre>    return &quot;\n&quot;.join(pages)</pre><p><strong>Important:</strong> Not every PDF contains selectable text. Scanned PDFs may require OCR before they can be processed. Document extraction quality directly affects retrieval quality.</p><p>After extraction, the text needs cleaning to remove unnecessary whitespace and formatting artifacts.</p><h3>4. Step 2 — Split documents into smaller chunks</h3><p>Why not directly send the entire HR policy document to an LLM?</p><p>Because large documents can contain more information than is necessary for a particular question. They also increase the amount of context sent to the model.</p><p>Instead, I divided the extracted text into smaller chunks.</p><p>My project configuration:</p><p>ParameterValueChunk size700 charactersChunk overlap100 charactersEmbedding modelDatabricks GTE Large ENEmbedding dimensions1024</p><p><strong>Why use overlap?</strong></p><p>Imagine a policy statement starts at the end of one chunk and continues in the next.</p><p>Without overlap, important context could be separated.</p><p>Overlap helps preserve continuity between neighboring chunks.</p><p>Chunking is not just a technical preprocessing step. It has a direct impact on the quality of retrieval.</p><p>A useful improvement for real-world projects is to experiment with different chunk sizes and evaluate retrieval quality rather than assuming one configuration works for every document.</p><h3>5. Step 3 — Convert text into embeddings</h3><p>Computers and vector search systems need a numerical representation of text to compare meaning.</p><p>This is where embeddings come in.</p><p>An embedding model converts a text chunk into a numerical vector.</p><p>For example:</p><pre>&quot;Employees can carry forward up to two<br>earned leave days.&quot;</pre><p>becomes a vector containing 1024 numerical dimensions in this project.</p><p>I used the Databricks GTE Large EN embedding endpoint.</p><p>Each processed record contains information such as:</p><ul><li>chunk_id</li><li>chunk_text</li><li>doc_name</li><li>embedding</li></ul><p>The embedding captures semantic information that can help identify related content even when the exact keywords differ.</p><h3>6. Step 4 — Store the embeddings in Delta Lake</h3><p>Generating embeddings is only one part of the pipeline. We also need to store them in a structured and reusable way.</p><p>I stored the processed document chunks and their embeddings in a Delta table:</p><p>peopleopsproject.genai_book.hr_docs_embeddings_v2</p><p>Why Delta Lake?</p><ul><li>Provides reliable storage for processed document data.</li><li>Integrates naturally with the Databricks ecosystem.</li><li>Supports downstream data processing and retrieval workflows.</li><li>Makes the embedding data reusable across application components.</li></ul><p>The resulting table becomes the source for the Vector Search index.</p><h3>7. Step 5 — Implement Databricks Vector Search</h3><p>Now comes the interesting part.</p><p>We have document chunks and their embeddings. How do we retrieve the most relevant information when an employee asks a question?</p><p>I used Databricks Vector Search.</p><p>Project configuration:</p><p>ComponentConfigurationEndpointpeopleops_vs_endpointIndexpeopleopsproject.genai_book.hr_docs_vs_index_v2Source tablehr_docs_embeddings_v2Embedding endpointdatabricks-gte-large-enSearch typeHybrid SearchRetrievalTop 3 chunks</p><h3>Why hybrid search?</h3><p>Semantic search focuses on meaning, while keyword search helps identify exact terms.</p><p>Consider a question:</p><p>“What is the EL carry-forward limit?”</p><p>The system should understand the meaning of the question while also recognizing important terms such as “EL” and “carry-forward”.</p><p>Hybrid search combines semantic and keyword-based retrieval to improve the chances of finding relevant policy content.</p><p>The retrieval process is:</p><ol><li>Employee submits a question.</li><li>The question is embedded.</li><li>Vector Search retrieves relevant document chunks.</li><li>The top results are returned.</li><li>Retrieved content becomes context for the LLM.</li><li>The LLM generates an answer using that context.</li></ol><h3>8. An important engineering lesson: document versions matter</h3><p>During this project, I included both a current leave policy and a superseded policy.</p><p>This introduces an important real-world challenge.</p><p>What happens if the vector index retrieves an older policy that contradicts the current one?</p><p>A good RAG application should not blindly trust every retrieved document.</p><p>A better approach is to maintain document metadata such as:</p><ul><li>Document name</li><li>Policy version</li><li>Effective date</li><li>Document status</li><li>Department or category</li></ul><p>Then apply appropriate filtering or ranking so that superseded documents do not influence answers about current policies.</p><p>This is an important distinction between building a simple RAG demo and designing a reliable enterprise knowledge assistant.</p><h3>9. What have we built so far?</h3><p>At this stage, we have completed the knowledge preparation and retrieval foundation.</p><p>We can:</p><ul><li>Process HR policy PDFs.</li><li>Extract and clean document content.</li><li>Split documents into manageable chunks.</li><li>Generate embeddings using GTE Large EN.</li><li>Store chunks and embeddings in Delta Lake.</li><li>Retrieve relevant information using Databricks Vector Search.</li></ul><p>But we still have a limitation.</p><p>Vector Search can retrieve policy information, but it cannot independently answer every employee-specific question.</p><p>For example:</p><ul><li>What is my remaining leave balance?</li><li>What is my employee ID?</li><li>Can you raise an HR support ticket?</li></ul><p>These require structured database access and an intelligent component that can decide which action to perform.</p><p>That’s where the AI agent comes in.</p><h3>10. What’s next?</h3><p>In Part 2, we’ll connect this RAG pipeline with structured SQL tools and build an intelligent HR assistant using LangGraph.</p><p>We’ll explore:</p><ul><li>Databricks GPT OSS 120B</li><li>LangGraph agent orchestration</li><li>SQL-based employee and leave tools</li><li>Human-in-the-loop ticket creation</li><li>Streamlit application</li><li>Databricks Apps deployment</li><li>MLflow tracing and evaluation</li></ul><p><a href="https://medium.com/@jc.prabhaswork/from-data-engineering-to-agentic-ai-building-an-intelligent-hr-assistant-with-databricks-b7494076acc3?postPublishedType=initial"><strong>Continue to Part 2:</strong> Building the Agentic AI Layer.</a></p><h3>Final thoughts</h3><p>One of my biggest takeaways from this implementation is that a useful GenAI application starts long before the LLM generates its first response.</p><p>Document quality, chunking strategy, embeddings, metadata, and retrieval quality all play an important role in the final answer.</p><p>If you’re learning Databricks, RAG, or Generative AI, I encourage you to build a small end-to-end implementation and experiment with each layer.</p><p><strong>Explore the complete implementation:</strong></p><p>GitHub: <a href="https://github.com/Prabhas92/PeopleOps_HR_Assistant">https://github.com/Prabhas92/PeopleOps_HR_Assistant</a></p><p>#Databricks #RAG #GenerativeAI #VectorSearch #DeltaLake #Python #DataEngineering #AIEngineering</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f2cf8ecdcb0d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Delta Lake? Moving Beyond the Problems of Traditional Data Lakes]]></title>
            <link>https://medium.com/@jc.prabhaswork/why-delta-lake-moving-beyond-the-problems-of-traditional-data-lakes-9a07fce412ff?source=rss-8778549674bb------2</link>
            <guid isPermaLink="false">https://medium.com/p/9a07fce412ff</guid>
            <category><![CDATA[delta-lake]]></category>
            <category><![CDATA[azure-data-lake]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Chethan Prabhas]]></dc:creator>
            <pubDate>Sat, 08 Aug 2026 10:12:10 GMT</pubDate>
            <atom:updated>2026-08-08T15:42:19.943Z</atom:updated>
            <content:encoded><![CDATA[<p>If you work in <strong>Data Engineering, Cloud, or Analytics</strong>, you have probably worked with a data lake.</p><p>You may have data sitting in:</p><ul><li><strong>Azure Data Lake Storage (ADLS)</strong></li><li><strong>Amazon S3</strong></li><li><strong>Google Cloud Storage (GCS)</strong></li></ul><p>And the data is usually stored as:</p><ul><li>Parquet</li><li>CSV</li><li>JSON</li><li>ORC</li></ul><p>At first, this looks perfect.</p><p>We have cheap storage, almost unlimited capacity, and tools like Spark can process huge amounts of data.</p><p>But as the data platform grows, a simple question starts to appear:</p><blockquote><strong><em>“If cloud storage is so good, why do we need Delta Lake?”</em></strong></blockquote><p>The answer becomes clear when we look at the problems we face with a traditional data lake.</p><p>In this article, we’ll understand:</p><ol><li>What a traditional data lake looks like</li><li>The problems we start facing in production</li><li>How Delta Lake solves these problems</li><li>The difference between a traditional data lake and Delta Lake</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fQD6q_5teEEbdOfaNOJv4w.png" /></figure><h3>1. What Do We Have Today? — Traditional Data Lake</h3><p>Let’s start with a simple architecture.</p><p>Imagine we receive data from different sources:</p><pre>        Data Sources<br>             |<br>             v<br>    +-------------------+<br>    | ADF / Spark /     |<br>    | Streaming Jobs    |<br>    +-------------------+<br>             |<br>             v<br>    +-------------------+<br>    | Cloud Data Lake   |<br>    | ADLS / S3 / GCS   |<br>    +-------------------+<br>             |<br>       Parquet / CSV<br>             |<br>             v<br>    +-------------------+<br>    | Spark / Trino /   |<br>    | Databricks / BI   |<br>    +-------------------+</pre><p>For example, our data lake might look like this:</p><pre>/orders/<br>    2026-08-01.parquet<br>    2026-08-02.parquet<br>    2026-08-03.parquet</pre><p>Spark can easily read these files:</p><pre>df = spark.read.parquet(&quot;/orders/&quot;)</pre><p>We can transform the data and write it back:</p><pre>df.write.parquet(&quot;/orders_processed/&quot;)</pre><p>Simple.</p><p>Cheap.</p><p>Scalable.</p><p>So what’s the problem?</p><p>The problem isn’t really <strong>storing</strong> the data.</p><p>The problem starts when we need our data lake to behave more like a <strong>reliable database table</strong>.</p><h3>2. Problem #1 — What Happens If a Job Fails Halfway?</h3><p>Imagine we have a Spark job that needs to write 100 Parquet files.</p><p>It starts writing:</p><pre>File 1   ✅<br>File 2   ✅<br>File 3   ✅<br>...<br>File 45  ✅<br>File 46  ❌</pre><p>The job fails.</p><p>But the first 45 files are already sitting in our data lake.</p><p>Now another process comes along and reads the directory.</p><p>It may see those 45 files and assume:</p><blockquote><em>“This is the complete dataset.”</em></blockquote><p>But it isn’t.</p><p>We have only a partial write.</p><p>This is one of the major problems with working directly with files.</p><p>We need a mechanism that says:</p><blockquote><strong><em>Either the entire operation succeeds, or the table should continue showing the previous valid state.</em></strong></blockquote><p>That’s where <strong>ACID transactions</strong> become important.</p><h3>3. Problem #2 — Updating a Single Record Isn’t Simple</h3><p>Let’s say we have a customer table:</p><pre>customer_id | name    | city<br>------------|---------|--------<br>101         | Rahul   | Delhi<br>102         | Chethan | Mumbai<br>103         | Prabhas | Chennai</pre><p>Tomorrow, Chethan moves from Mumbai to Bangalore.</p><p>In a database, we can simply write:</p><pre>UPDATE customers<br>SET city = &#39;Bangalore&#39;<br>WHERE customer_id = 102;</pre><p>Done.</p><p>But a Parquet file works differently.</p><p>The data is stored inside files.</p><p>We can’t simply open the Parquet file and change one row like we would in a database.</p><p>Typically, the affected data needs to be rewritten.</p><p>For a small dataset, that’s manageable.</p><p>But imagine a table containing:</p><p><strong>10 TB of data.</strong></p><p>Changing a handful of records shouldn’t require us to think about rewriting massive amounts of data.</p><p>This is where traditional data lakes become inconvenient.</p><h3>4. Problem #3 — Deletes Are Difficult Too</h3><p>The same problem exists with deletes.</p><p>Suppose a user asks us to remove their data.</p><p>In a database:</p><pre>DELETE FROM customers<br>WHERE customer_id = 102;</pre><p>Simple.</p><p>With files, the record is physically stored inside a file.</p><p>We need additional processing to identify the affected files, rewrite the data without those records, and manage the old files.</p><p>For large datasets, this becomes more complicated.</p><h3>5. Problem #4 — What If the Schema Changes?</h3><p>Data changes over time.</p><p>Imagine our customer data originally looks like this:</p><pre>customer_id<br>name<br>city</pre><p>Everything works.</p><p>Then the source application team adds:</p><pre>email</pre><p>Now we receive:</p><pre>customer_id<br>name<br>city<br>email</pre><p>This is a normal business requirement.</p><p>But what happens if the source system accidentally changes:</p><pre>customer_id = 101</pre><p>from an integer into:</p><pre>customer_id = &quot;ABC&quot;</pre><p>Or suddenly changes:</p><pre>amount = 100.50</pre><p>into:</p><pre>amount = &quot;ONE HUNDRED&quot;</pre><p>If our data lake doesn’t have proper schema controls, bad data can make its way downstream.</p><p>And eventually:</p><pre>Bad Source Data<br>       ↓<br>Data Lake<br>       ↓<br>Transformation Failure<br>       ↓<br>Dashboard Failure<br>       ↓<br>Business User: &quot;Why is the report broken?&quot;</pre><p>This is where <strong>schema enforcement</strong> becomes valuable.</p><h3>6. Problem #5 — The Small File Problem</h3><p>This is a very common issue in streaming and incremental pipelines.</p><p>Imagine we receive small batches of data every minute.</p><p>Our pipeline keeps writing them:</p><pre>part-0001.parquet<br>part-0002.parquet<br>part-0003.parquet<br>...<br>part-10000.parquet</pre><p>After some time, we may have thousands or even millions of small files.</p><p>Instead of:</p><pre>100 large files</pre><p>we now have:</p><pre>1,000,000 tiny files</pre><p>When Spark or another query engine reads this data, it has to deal with all those files.</p><p>That means:</p><ul><li>More file metadata</li><li>More file opening operations</li><li>More scheduling overhead</li><li>Slower queries</li><li>Poorer performance</li></ul><p>This is commonly called the <strong>Small File Problem</strong>.</p><h3>7. Problem #6 — Where Did This Data Come From?</h3><p>Now imagine someone asks:</p><blockquote><em>“Yesterday the customer count was 10 million. Today it is 9.5 million. What happened?”</em></blockquote><p>With a simple collection of Parquet files, answering this question isn’t always easy.</p><p>We may need:</p><ul><li>Backup copies</li><li>Snapshots</li><li>External audit logs</li><li>Custom versioning logic</li></ul><p>Without proper version history, understanding how the dataset changed can become difficult.</p><p>And if someone accidentally overwrites or deletes data, recovering the previous state can be even harder.</p><h3>8. So What Do We Actually Need?</h3><p>At this point, our requirements are becoming clear.</p><p>We want the benefits of a data lake:</p><pre>✓ Cheap cloud storage<br>✓ Huge scale<br>✓ Flexible storage<br>✓ Parquet<br>✓ Spark processing</pre><p>But we also want database-like capabilities:</p><pre>✓ Reliable transactions<br>✓ Updates<br>✓ Deletes<br>✓ MERGE / Upserts<br>✓ Schema enforcement<br>✓ Schema evolution<br>✓ Version history<br>✓ Better file management</pre><p>So the question becomes:</p><blockquote><strong><em>Can we get both?</em></strong></blockquote><p>Yes.</p><p>This is where <strong>Delta Lake</strong> comes in.</p><h3>9. Enter Delta Lake</h3><p>Delta Lake is a <strong>storage layer</strong> that sits on top of cloud object storage.</p><p>It doesn’t replace:</p><ul><li>ADLS</li><li>S3</li><li>GCS</li></ul><p>And it doesn’t stop using Parquet.</p><p>Instead, Delta Lake adds another important component:</p><h3>The Transaction Log</h3><p>Conceptually:</p><pre>              Delta Table<br>                   |<br>        +----------+----------+<br>        |                     |<br>        v                     v<br>  Parquet Files          _delta_log<br>    (Data)             (Transaction Log)<br>        |                     |<br>        +----------+----------+<br>                   |<br>                   v<br>             Cloud Storage</pre><p>The Parquet files contain the actual data.</p><p>The _delta_log keeps track of changes made to the table.</p><p>This transaction log is one of the key ideas behind Delta Lake.</p><h3>10. How Does the Transaction Log Help?</h3><p>Imagine we have a Delta table:</p><pre>customers/</pre><p>Inside it, we may have:</p><pre>customers/<br>    part-0001.parquet<br>    part-0002.parquet<br>    part-0003.parquet</pre><pre>    _delta_log/<br>        00000000000000000000.json<br>        00000000000000000001.json<br>        00000000000000000002.json</pre><p>The transaction log records changes to the table.</p><p>Conceptually:</p><pre>Version 0<br>    ↓<br>Files A, B, C added</pre><pre>Version 1<br>    ↓<br>File D added</pre><pre>Version 2<br>    ↓<br>File A removed<br>File E added</pre><pre>Version 3<br>    ↓<br>Customer record updated</pre><p>So instead of simply looking at:</p><blockquote><em>“Which files are sitting in this folder?”</em></blockquote><p>Delta can understand:</p><blockquote><strong><em>“What is the current valid state of this table?”</em></strong></blockquote><p>That difference is extremely powerful.</p><h3>What’s Next?</h3><p>So far, we have looked at <strong>why Delta Lake is needed</strong> and how it addresses some of the major problems we face with traditional data lakes.</p><p>But this is only the beginning.</p><p>We have seen the <strong>_delta_log</strong> and understood that it plays a key role in maintaining the state of a Delta table.</p><p>But what exactly is stored inside it?</p><p>What happens when we <strong>INSERT, UPDATE, DELETE, or MERGE</strong> data?</p><p>How does Delta Lake maintain <strong>ACID transactions</strong>?</p><p>How does it handle <strong>concurrent writes</strong> from multiple pipelines?</p><p>How does <strong>Time Travel</strong> work?</p><p>What happens when the schema changes?</p><p>And what about <strong>small files, OPTIMIZE, VACUUM, Z-Ordering, Liquid Clustering, Deletion Vectors, and Delta Table Cloning</strong>?</p><p>There is a lot more happening behind the scenes than simply storing data in Parquet files.</p><h3>🚀 This is where the Delta Lake journey begins.</h3><p>In this series, we will gradually uncover the <strong>internals, features, performance optimizations, and practical use cases of Delta Lake</strong> — starting from the basics and going all the way to advanced concepts.</p><p>The goal is not just to learn <strong>what Delta Lake can do</strong>, but to understand <strong>how it works and why we need it</strong>.</p><p><strong>Stay tuned for the next part, where we’ll open up the </strong><strong>_delta_log and see what really happens behind the scenes when data changes.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9a07fce412ff" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Azure Data Factory Incremental Data Loading Series]]></title>
            <link>https://medium.com/@jc.prabhaswork/azure-data-factory-incremental-data-loading-series-a2f41100ea6c?source=rss-8778549674bb------2</link>
            <guid isPermaLink="false">https://medium.com/p/a2f41100ea6c</guid>
            <category><![CDATA[incremental-load]]></category>
            <category><![CDATA[azure-data-factory]]></category>
            <dc:creator><![CDATA[Chethan Prabhas]]></dc:creator>
            <pubDate>Mon, 03 Aug 2026 04:52:41 GMT</pubDate>
            <atom:updated>2026-08-14T11:31:53.705Z</atom:updated>
            <content:encoded><![CDATA[<h3>Which Incremental Loading Strategy Should You Choose?</h3><blockquote><strong><em>Series:</em></strong><em> Azure Data Factory Incremental Data Loading Series<br></em><strong><em>Level:</em></strong><em> Beginner → Advanced</em></blockquote><h3>Welcome 👋</h3><p>Congratulations! If you’ve followed this series, you’ve learned three<br>different approaches to Incremental Loading in Azure Data Factory.</p><p>This article compares them and helps you choose the right one.</p><h3>Evolution</h3><pre>JSON Watermark<br>      │<br>      ▼<br>Watermark Table<br>      │<br>      ▼<br>Enterprise Framework</pre><h3>Comparison Table</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qdWXDJoiqpOxB0xjbWL8Kg.png" /></figure><h3>Decision Guide</h3><h3>JSON Watermark</h3><p>Use when: — Learning ADF — Building POCs — One or two tables</p><h3>Watermark Table</h3><p>Use when: — Multiple tables — Reusable pipelines — Metadata-driven<br>framework</p><h3>Enterprise Framework</h3><p>Use when: — Production systems — Auditing required — Monitoring<br>required — Enterprise support</p><h3>Decision Tree</h3><pre>Learning ADF? <br>      │<br> Yes  ┴  No<br> │       │<br> ▼       ▼<br>JSON   Multiple Tables?<br>           │<br>      No ──┴── Yes<br>      │        │<br>      ▼        ▼<br>   JSON    Need Auditing?<br>                │<br>         No ────┴──── Yes<br>          │            │<br>          ▼            ▼<br>  Watermark Table  Enterprise Framework</pre><h3>Final Thoughts</h3><p>Choose the solution that matches your project, not the most complex one.</p><ul><li>JSON Watermark → Learn fundamentals.</li><li>Watermark Table → Scale projects.</li><li>Enterprise Framework → Production-ready pipelines.</li></ul><p>As Data Engineers, our responsibility isn’t to build the most complicated solution. Our responsibility is to build the <strong>right solution</strong> for the business.</p><p>I hope this four-part series has helped you understand not only <strong>how</strong> to implement Incremental Loading in Azure Data Factory, but also <strong>why</strong> each design exists and <strong>when</strong> to use it.</p><p>Thank you for reading!</p><p>Happy Learning!</p><p>📚 Azure Data Factory Incremental Data Loading Series</p><p>✅ <a href="https://medium.com/@jc.prabhaswork/azure-data-factory-incremental-data-loading-series-d23921dfff28">Part 1</a> — Incremental Loading using JSON Watermark Files</p><p>✅ <a href="https://medium.com/@jc.prabhaswork/azure-data-factory-incremental-data-loading-series-98fa8399da19">Part 2</a> — Incremental Loading using Watermark Tables</p><p>✅ <a href="https://medium.com/@jc.prabhaswork/azure-data-factory-incremental-data-loading-series-7b739c6b9c45">Part 3</a> — Enterprise Incremental Framework with Auditing</p><p>✅ Part 4 — JSON vs Watermark Table vs Enterprise Framework</p><h3>🔗 GitHub Repository</h3><p>For the <strong>complete templates, ADF pipeline configurations, expressions, and source code</strong>, please refer to the GitHub repository below:</p><p>👉 <a href="https://github.com/Prabhas92/ADFProject"><strong>View the ADF Project on GitHub</strong></a></p><p>Feel free to explore, reuse, and customize the templates for your own Azure Data Factory projects.</p><p><strong><em>See you in the next Azure Data Engineering series! 😊</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a2f41100ea6c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Azure Data Factory Incremental Data Loading Series]]></title>
            <link>https://medium.com/@jc.prabhaswork/azure-data-factory-incremental-data-loading-series-7b739c6b9c45?source=rss-8778549674bb------2</link>
            <guid isPermaLink="false">https://medium.com/p/7b739c6b9c45</guid>
            <category><![CDATA[adf-copy-activity]]></category>
            <category><![CDATA[incremental-load]]></category>
            <category><![CDATA[azure-data-factory]]></category>
            <category><![CDATA[adls-gen2]]></category>
            <dc:creator><![CDATA[Chethan Prabhas]]></dc:creator>
            <pubDate>Mon, 03 Aug 2026 04:24:08 GMT</pubDate>
            <atom:updated>2026-08-14T11:35:58.077Z</atom:updated>
            <content:encoded><![CDATA[<h3>Part 3 — Enterprise Incremental Framework with Auditing</h3><blockquote><strong><em>Series:</em></strong><em> Azure Data Factory Incremental Data Loading Series<br></em><strong><em>Part:</em></strong><em> 3 of 4</em></blockquote><h3>Welcome Back 👋</h3><p>If you’ve reached this article, congratulations!</p><p>In <a href="https://medium.com/@jc.prabhaswork/azure-data-factory-incremental-data-loading-series-d23921dfff28"><strong>Part 1</strong></a>, we built our first incremental loading pipeline using a<br>JSON watermark file.</p><p>In <a href="https://medium.com/@jc.prabhaswork/azure-data-factory-incremental-data-loading-series-98fa8399da19"><strong>Part 2</strong></a>, we improved that design by replacing JSON files with a<br>centralized Watermark Table.</p><p>At this point our solution works well.</p><p>But here’s the question every production support engineer eventually<br>asks:</p><blockquote><strong><em>“How do I know what happened after the pipeline finishes?”</em></strong></blockquote><p>This chapter is about answering that question.</p><h3>The Business Problem</h3><p>Imagine it’s <strong>2:30 AM</strong>.</p><p>Your overnight pipeline has failed.</p><p>At 9:00 AM your manager asks:</p><ul><li>Which table failed?</li><li>When did it fail?</li><li>How many rows were copied before failure?</li><li>Which watermark was used?</li><li>Which watermark should be used on the next run?</li></ul><p>If your only answer is:</p><blockquote><em>“I’ll check Azure Data Factory Monitor…”</em></blockquote><p>you’re already in trouble.</p><p>ADF Monitor is useful, but enterprise applications usually require<br><strong>their own audit trail</strong>.</p><h3>Why Part 2 Is Still Not Enough</h3><p>The Watermark Table solved one problem:</p><p>✅ Tracking the last successful load.</p><p>But it doesn’t answer:</p><ul><li>How long did the pipeline run?</li><li>Did it succeed or fail?</li><li>Which table took the longest?</li><li>Which execution copied the most rows?</li></ul><p>Enterprise teams need these answers every day.</p><h3>What Is an Enterprise Framework?</h3><p>Think of a production pipeline like an airplane.</p><p>The goal isn’t just to fly.</p><p>It also needs a <strong>black box</strong>.</p><p>Even if something goes wrong, engineers should know exactly what<br>happened.</p><p>Our pipeline should do the same.</p><h3>Technologies Used</h3><ul><li>Azure Data Factory</li><li>Azure SQL Database</li><li>Watermark Table</li><li>Lookup Activity</li><li>ForEach Activity</li><li>If Condition</li><li>Pipeline Variables</li></ul><h3>Pipeline Overview</h3><p>Our production pipeline follows this sequence:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nsHFQXjhUJayen1EZ-aKbA.png" /><figcaption>Incremental Loading with Logging</figcaption></figure><h3>Activity 1 — Capture Pipeline Start Time</h3><h3>Why do we need this?</h3><p>Imagine running a marathon.</p><p>Would you know how long it took if you never looked at the clock before<br>starting?</p><p>Exactly.</p><p>The same idea applies to pipelines.</p><p>Before processing any table, we capture the current UTC time.</p><p>Later this helps calculate:</p><ul><li>Execution Duration</li><li>Start Time</li><li>End Time</li><li>SLA Monitoring</li></ul><h3>Example Variable</h3><pre>PipelineStartTime<br>2026-06-21T02:00:00Z</pre><h3>Best Practice</h3><p>Always store timestamps in <strong>UTC</strong>.</p><h3>Activity 2 — Read Metadata</h3><p>Instead of hardcoding table names, we keep metadata in one place.</p><p>Example:</p><pre>[<br>  {<br>    &quot;TableName&quot;:&quot;FactBookings&quot;,<br>    &quot;Schema&quot;:&quot;dbo&quot;,<br>    &quot;WatermarkColumn&quot;:&quot;booking_date&quot;<br>  }<br>]</pre><h3>Why?</h3><p>Tomorrow your project may contain:</p><ul><li>Customers</li><li>Flights</li><li>Payments</li><li>Bookings</li><li>Airports</li></ul><p>One pipeline should handle all of them.</p><h3>Activity 3 — ForEach</h3><p>The ForEach activity loops through every metadata record.</p><p>Instead of creating ten Copy Activities,</p><p>we create <strong>one reusable workflow</strong>.</p><p>Think of it like reading a to-do list.</p><pre>Table 1<br>↓<br>Table 2<br>↓<br>Table 3</pre><p>The same logic repeats for every table.</p><h3>Activity 4 — Lookup Old Watermark</h3><p>This activity reads the previous successful watermark.</p><h3>SQL Query</h3><pre>SELECT WatermarkValue<br>FROM dbo.WatermarkTable<br>WHERE TableName=&#39;FactBookings&#39;;</pre><h3>Why?</h3><p>Without knowing the previous watermark, the pipeline has no idea where<br>to continue.</p><p>Think of this as opening a bookmark before reading the next chapter of a<br>book.</p><h3>Sample Output</h3><pre>{<br>  &quot;firstRow&quot;: {<br>    &quot;WatermarkValue&quot;:&quot;2026-06-20 08:00:00&quot;<br>  }<br>}</pre><p>ADF can access it using:</p><pre>@activity(&#39;Lookup_OldWatermark&#39;).output.firstRow.WatermarkValue</pre><h3>Activity 5 — Lookup Latest Watermark</h3><p>Now we ask SQL Server one simple question.</p><blockquote><em>“What is the latest record currently available?”</em></blockquote><h3>SQL Query</h3><pre>SELECT MAX(booking_date) AS LatestWatermark<br>FROM dbo.FactBookings;</pre><h3>Why MAX()?</h3><p>We only need the newest timestamp.</p><p>Example:</p><pre>Old Watermark<br>2026-06-20 08:00<br>Latest Record<br>2026-06-20 10:30</pre><p>Everything between these two timestamps becomes the next incremental<br>batch.</p><h3>Activity 6 — If Condition</h3><p>Now the pipeline compares both values.</p><pre>Old Watermark<br>08:00<br>Latest Watermark<br>10:30</pre><p>If they are different,</p><p>there is new data.</p><p>If they are equal,</p><p>there is nothing to copy.</p><p>This simple comparison saves compute time and storage costs.</p><h3>Activity 7 — Copy Incremental Data</h3><p>This is the heart of the pipeline.</p><p>Instead of copying the entire table, we copy only the records between<br>the previous and latest watermark.</p><h3>SQL Query</h3><pre>SELECT *<br>FROM dbo.FactBookings<br>WHERE booking_date &gt; @OldWatermark<br>AND booking_date &lt;= @LatestWatermark;</pre><h3>Why &gt;?</h3><p>Using &gt; avoids copying the last successful record again.</p><h3>Why &lt;=?</h3><p>Using &lt;= ensures the newest record is included.</p><h3>Dynamic Expressions</h3><p>Old Watermark</p><pre>@activity(&#39;Lookup_OldWatermark&#39;).output.firstRow.WatermarkValue</pre><p>Latest Watermark</p><pre>@activity(&#39;Lookup_LatestWatermark&#39;).output.firstRow.LatestWatermark</pre><p>ADF replaces these expressions at runtime before executing the SQL<br>query.</p><h3>Activity 8 — Update Watermark</h3><p>Once the Copy Activity succeeds, the watermark table must be updated.</p><h3>SQL</h3><pre>UPDATE dbo.WatermarkTable<br>SET WatermarkValue=@LatestWatermark<br>WHERE TableName=&#39;FactBookings&#39;;</pre><h3>Why after Copy?</h3><p>If the copy fails and the watermark is updated first, the next execution<br>skips records permanently.</p><p>Always follow this order:</p><pre>Copy Activity<br>      │<br>      ▼<br>Update Watermark</pre><h3>Activity 9 — Script Activity (Audit Logging)</h3><p>Production pipelines should leave behind an execution history.</p><p>Instead of wondering what happened, engineers should be able to query an<br>audit table.</p><h3>Sample Audit Table</h3><pre>CREATE TABLE dbo.PipelineAudit<br>(<br> AuditID INT IDENTITY,<br> PipelineName VARCHAR(100),<br> TableName VARCHAR(100),<br> RowsCopied INT,<br> OldWatermark DATETIME,<br> NewWatermark DATETIME,<br> Status VARCHAR(20),<br> StartTime DATETIME,<br> EndTime DATETIME,<br> ErrorMessage NVARCHAR(MAX)<br>);</pre><h3>Success Audit</h3><pre>INSERT INTO dbo.PipelineAudit<br>(<br>PipelineName,<br>TableName,<br>RowsCopied,<br>OldWatermark,<br>NewWatermark,<br>Status,<br>StartTime,<br>EndTime<br>)<br>VALUES<br>(<br>&#39;IncrementalPipeline&#39;,<br>&#39;FactBookings&#39;,<br>2350,<br>&#39;2026-06-20 08:00&#39;,<br>&#39;2026-06-20 10:30&#39;,<br>&#39;Success&#39;,<br>GETUTCDATE(),<br>GETUTCDATE()<br>);</pre><h3>Failure Audit</h3><pre>INSERT INTO dbo.PipelineAudit<br>(<br>PipelineName,<br>TableName,<br>Status,<br>ErrorMessage<br>)<br>VALUES<br>(<br>&#39;IncrementalPipeline&#39;,<br>&#39;FactBookings&#39;,<br>&#39;Failed&#39;,<br>&#39;Primary Key Violation&#39;<br>);</pre><h3>Why Auditing Matters</h3><p>Without an audit table, troubleshooting depends on Azure Data Factory<br>Monitor.</p><p>With an audit table, support engineers can answer:</p><ul><li>Which table failed?</li><li>How many rows were copied?</li><li>How long did the pipeline take?</li><li>What was the error?</li></ul><p>within seconds.</p><h3>Best Practices</h3><ul><li>Update watermark only after successful copy.</li><li>Log every execution.</li><li>Record row counts.</li><li>Store start and end times.</li><li>Capture error messages.</li><li>Never delete audit history.</li></ul><h3>Common Mistakes</h3><p>❌ Hardcoding table names</p><p>❌ Returning multiple watermark rows</p><p>❌ Comparing local time instead of UTC</p><p>❌ Skipping metadata-driven design</p><h3>Conclusion</h3><p>Congratulations! 🎉</p><p>You now have a production-ready incremental loading framework.</p><p>By combining:</p><ul><li>Watermark Tables</li><li>Dynamic SQL</li><li>Copy Activity</li><li>Script Activity</li><li>Audit Logging</li></ul><p>you’ve built a solution that is scalable, maintainable and easier to<br>troubleshoot.</p><p>In <a href="https://medium.com/@jc.prabhaswork/azure-data-factory-incremental-data-loading-series-a2f41100ea6c"><em>Part 4</em></a>, we’ll compare all three incremental loading approaches and<br>discuss when each one should be used.</p><h3>🔗 GitHub Repository</h3><p>For the <strong>complete templates, ADF pipeline configurations, expressions, and source code</strong>, please refer to the GitHub repository below:</p><p>👉 <a href="https://github.com/Prabhas92/ADFProject"><strong>View the ADF Project on GitHub</strong></a></p><p>Feel free to explore, reuse, and customize the templates for your own Azure Data Factory projects.</p><p>Happy Learning! 🚀</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7b739c6b9c45" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Azure Data Factory Incremental Data Loading Series]]></title>
            <link>https://medium.com/@jc.prabhaswork/azure-data-factory-incremental-data-loading-series-98fa8399da19?source=rss-8778549674bb------2</link>
            <guid isPermaLink="false">https://medium.com/p/98fa8399da19</guid>
            <category><![CDATA[adls-gen2]]></category>
            <category><![CDATA[azure-foreach]]></category>
            <category><![CDATA[incremental-load]]></category>
            <category><![CDATA[azure-data-factory]]></category>
            <dc:creator><![CDATA[Chethan Prabhas]]></dc:creator>
            <pubDate>Sat, 01 Aug 2026 14:19:10 GMT</pubDate>
            <atom:updated>2026-08-14T11:37:53.322Z</atom:updated>
            <content:encoded><![CDATA[<h3>Part 2 — Incremental Loading using Watermark Tables</h3><blockquote><strong><em>Series:</em></strong><em> Azure Data Factory Incremental Data Loading Series<br></em><strong><em>Level:</em></strong><em> Intermediate<br></em><strong><em>Project Type:</em></strong><em> Real-world Azure Data Factory Pipeline</em></blockquote><h3>Welcome Back 👋</h3><p>In <a href="https://medium.com/@jc.prabhaswork/azure-data-factory-incremental-data-loading-series-d23921dfff28"><strong>Part 1</strong></a>, we learned how to implement incremental loading using a<br>simple JSON watermark file.</p><p>That approach is perfect for learning, but imagine your company has:</p><ul><li>50 tables</li><li>100 tables</li><li>500 tables</li></ul><p>Would you really maintain 500 JSON files?</p><p>Probably not.</p><p>This is where <strong>Watermark Tables</strong> come into the picture.</p><p>Instead of storing the last successful load in separate JSON files, we<br>store all watermarks inside a single SQL table. This is how many<br>real-world Azure Data Factory projects begin their journey toward a<br>metadata-driven framework.</p><h3>Business Problem</h3><p>Our company has multiple source tables.</p><p>Instead of building one pipeline per table, we want:</p><ul><li>One reusable pipeline</li><li>One watermark table</li><li>One metadata list</li><li>Automatic processing of multiple tables</li></ul><h3>From JSON to Watermark Table</h3><h3>Previous Approach</h3><pre>Table A -&gt; lastload.json<br>Table B -&gt; lastload.json<br>Table C -&gt; lastload.json</pre><p>Lots of files to maintain.</p><h3>New Approach</h3><pre>Watermark Table</pre><pre>+-------------+----------------------+<br>| TableName   | WatermarkValue       |<br>+-------------+----------------------+<br>| Customers   | 2026-06-20 10:00:00  |<br>| Orders      | 2026-06-20 11:15:00  |<br>| Products    | 2026-06-20 09:45:00  |<br>+-------------+----------------------+</pre><p>One table. Many pipelines. Easy to manage.</p><h3>Solution Architecture</h3><pre>Pipeline<br>   │<br>   ▼<br>ForEach (Table List)<br>   │<br>   ├── Read Old Watermark<br>   ├── Read Latest Watermark<br>   ├── Compare Values<br>   ├── Copy Incremental Data<br>   └── Update Watermark Table</pre><h3>Technologies Used</h3><ul><li>Azure Data Factory</li><li>SQL Server</li><li>Watermark Table</li><li>Lookup Activity</li><li>If Condition</li><li>Copy Activity</li><li>Stored Procedure</li><li>Parameterized Datasets</li><li>Dynamic Expressions</li></ul><h3>Pipeline Flow</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lwD_CRrMBOUWve5KMcLvmw.png" /><figcaption>Incremental Loading Using Watermark Table</figcaption></figure><h3>Understanding the Pipeline</h3><h3>Step 1 — Read Table Metadata</h3><p>Instead of hardcoding table names, the pipeline receives an array like:</p><pre>[<br> {<br>   &quot;TableName&quot;:&quot;iplteams&quot;,<br>   &quot;TableSchemaName&quot;:&quot;dbo&quot;,<br>   &quot;WaterMark_Column&quot;:&quot;last_updated&quot;<br> }<br>]</pre><p>Think of this as a to-do list for the pipeline.</p><h3>Step 2 — ForEach Activity</h3><p>The pipeline loops through every table.</p><p>Instead of creating ten Copy Activities, we create <strong>one reusable Copy<br>Activity</strong>.</p><p>This is one of the biggest improvements over beginner pipelines.</p><h3>Step 3 — Lookup Old Watermark</h3><p>The first Lookup reads the previous successful timestamp.</p><pre>SELECT WatermarkValue<br>FROM watermarktable<br>WHERE TableName=&#39;iplteams&#39;</pre><p>This tells us where the previous run stopped.</p><h3>Step 4 — Lookup Latest Watermark</h3><p>The second Lookup finds the newest record in the source table.</p><pre>SELECT MAX(last_updated)<br>FROM dbo.iplteams</pre><p>Now we know the latest available data.</p><h3>Step 5 — If Condition</h3><p>The pipeline asks a simple question:</p><blockquote><em>“Has anything changed since the last successful run?”</em></blockquote><p>If the latest watermark is greater than the previous one, the pipeline<br>continues.</p><p>Otherwise, it skips the table.</p><p>This saves unnecessary work.</p><h3>Step 6 — Copy Incremental Data</h3><p>ADF builds the SQL query dynamically.</p><pre>SELECT *<br>FROM dbo.iplteams<br>WHERE last_updated &gt; OldWatermark<br>AND last_updated &lt;= NewWatermark</pre><p>Notice that the table name and watermark column are dynamic.</p><p>The same Copy Activity works for every table.</p><h3>Step 7 — Update the Watermark</h3><p>Only after the copy succeeds do we update the watermark table.</p><p>This is extremely important.</p><p>If the copy fails, we do <strong>not</strong> update the watermark.</p><p>Otherwise, new records could be skipped forever.</p><h3>Why This Design is Better</h3><p>Compared to JSON watermark files:</p><p>✅ One watermark table</p><p>✅ Supports many source tables</p><p>✅ Easier maintenance</p><p>✅ Reusable pipeline</p><p>✅ Dynamic SQL</p><p>✅ Metadata-driven design</p><h3>Runtime Execution</h3><pre>Pipeline Starts<br>      │<br>Read Table List<br>      │<br>ForEach<br>      │<br>Read Old Watermark<br>      │<br>Read Latest Watermark<br>      │<br>Compare<br>      │<br>Copy Incremental Data<br>      │<br>Update Watermark<br>      │<br>Next Table<br>      │<br>Pipeline Completed</pre><h3>Real-World Analogy</h3><p>Imagine a library.</p><p>Instead of keeping one sticky note inside every book to remember the<br>last page you read, you maintain a notebook.</p><p>Each page contains:</p><ul><li>Book Name</li><li>Last Page Read</li></ul><p>Whenever you finish reading a book, you update just one row in the<br>notebook.</p><p>That’s exactly what a Watermark Table does.</p><h3>Common Mistakes</h3><p>❌ Hardcoding table names</p><p>❌ Updating watermark before copy</p><p>❌ Forgetting to parameterize datasets</p><p>❌ Ignoring empty tables</p><p>❌ Using separate pipelines for every table</p><h3>Production Tips</h3><ul><li>Keep watermark updates transactional.</li><li>Store table metadata separately if many tables exist.</li><li>Use parameterized datasets and dynamic expressions.</li><li>Add logging and monitoring.</li></ul><p>We’ll cover these in <a href="https://medium.com/@jc.prabhaswork/azure-data-factory-incremental-data-loading-series-7b739c6b9c45"><em>Part 3</em></a>.</p><h3>Key Takeaways</h3><ul><li>A Watermark Table centralizes state management.</li><li>One pipeline can process many tables.</li><li>Dynamic expressions reduce duplication.</li><li>The design is scalable and maintainable.</li><li>This is a common enterprise pattern.</li></ul><h3>Conclusion</h3><p>Congratulations! 🎉</p><p>You’ve moved beyond the beginner approach.</p><p>Instead of maintaining multiple JSON files, you now have a reusable<br>framework that can process multiple tables using a single Azure Data<br>Factory pipeline.</p><p>This approach introduces two important engineering concepts:</p><ul><li>Metadata-driven design</li><li>Reusable pipelines</li></ul><p>In the <a href="https://medium.com/@jc.prabhaswork/azure-data-factory-incremental-data-loading-series-7b739c6b9c45"><em>next article</em></a>, we’ll take one more step toward production by<br>adding <strong>audit logging, Script Activities, monitoring, and error<br>handling</strong> to build an enterprise-ready incremental loading framework.</p><h3>🔗 GitHub Repository</h3><p>For the <strong>complete templates, ADF pipeline configurations, expressions, and source code</strong>, please refer to the GitHub repository below:</p><p>👉 <a href="https://github.com/Prabhas92/ADFProject"><strong>View the ADF Project on GitHub</strong></a></p><p>Feel free to explore, reuse, and customize the templates for your own Azure Data Factory projects.</p><p>Happy Learning! 🚀</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=98fa8399da19" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Azure Data Factory Incremental Data Loading Series]]></title>
            <link>https://medium.com/@jc.prabhaswork/azure-data-factory-incremental-data-loading-series-d23921dfff28?source=rss-8778549674bb------2</link>
            <guid isPermaLink="false">https://medium.com/p/d23921dfff28</guid>
            <category><![CDATA[adf-copy-activity]]></category>
            <category><![CDATA[incremental-loading]]></category>
            <category><![CDATA[azure-data-factory]]></category>
            <dc:creator><![CDATA[Chethan Prabhas]]></dc:creator>
            <pubDate>Sat, 01 Aug 2026 13:33:44 GMT</pubDate>
            <atom:updated>2026-08-14T11:37:51.373Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><strong><em>A practical series built from real Azure Data Factory projects</em></strong></blockquote><p>Welcome! 👋</p><p>If you’re learning Azure Data Factory (ADF), you’ve probably come across<br>the term <strong>Incremental Loading</strong>. It sounds simple, but once you start<br>building real pipelines, you’ll realize there are multiple ways to<br>implement it.</p><p>This series is designed to take you from the simplest implementation to<br>an enterprise-ready solution using real projects that I built while<br>learning Azure Data Engineering.</p><p>Whether you’re preparing for interviews, building portfolio projects, or<br>working on production pipelines, this series will help you understand<br>not only <strong>how</strong> to build incremental pipelines, but <strong>why</strong> they are<br>designed this way.</p><h3>📚 Series Roadmap</h3><h3>Part 1 — Incremental Loading using JSON Watermark Files (Beginner)</h3><p>Learn the easiest way to build an incremental pipeline using a JSON file<br>to store the last successful load timestamp.</p><p>You’ll learn: — What Incremental Loading is — Why Full Load is<br>inefficient — JSON Watermark concept — Lookup Activity — Copy Activity -<br>Updating the watermark — Advantages &amp; limitations</p><h3>Part 2 — Incremental Loading using Watermark Tables (Intermediate)</h3><p>Scale the design using a database watermark table.</p><p>You’ll learn: — Metadata-driven pipelines — Watermark table design -<br>ForEach — Dynamic SQL — Multiple table processing</p><h3>Part 3 — Enterprise Incremental Framework with Auditing (Advanced)</h3><p>Build a production-ready framework.</p><p>You’ll learn: — Audit logging — Script Activity — Error handling -<br>Monitoring — Production best practices</p><h3>Part 4 — Which Incremental Loading Strategy Should You Choose?</h3><p>A practical comparison of all three methods.</p><h3>Part 1 — Incremental Loading using JSON Watermark Files</h3><p><strong>Series:</strong> Azure Data Factory Incremental Data Loading Series</p><p><strong>Difficulty:</strong> Beginner</p><h3>Why should you care?</h3><p>Imagine you work for an airline company.</p><p>Every day, customers book flights through an online portal. These<br>bookings are stored in an Azure SQL Database.</p><p>Yesterday, your table contained <strong>10 million records</strong>.</p><p>Today, only <strong>4,500 new bookings</strong> were added.</p><p>Now ask yourself one question.</p><blockquote><em>Should we copy all 10 million records again just to get those 4,500<br>new rows?</em></blockquote><p>Probably not.</p><p>Doing a full load every day wastes:</p><ul><li>Time</li><li>Compute</li><li>Storage</li><li>Money</li></ul><p>A smarter approach is to copy <strong>only the new records</strong>.</p><p>This is called <strong>Incremental Data Loading</strong>.</p><h3>The Business Problem</h3><p>Our reporting team needs fresh booking data every day.</p><p>The pipeline should:</p><ul><li>Copy only newly added records</li><li>Avoid duplicates</li><li>Run faster than a full load</li><li>Be easy to understand</li></ul><h3>Our Solution</h3><p>Instead of remembering everything, the pipeline remembers just <strong>one<br>thing</strong>:</p><blockquote><em>“What was the last record I copied successfully?”</em></blockquote><p>We store this value inside a small JSON file called <strong>lastload.json</strong>.</p><p>Example:</p><pre>{<br>  &quot;lastload&quot;: &quot;2026-06-20 12:00:00&quot;<br>}</pre><p>This timestamp becomes our <strong>watermark</strong>.</p><h3>Solution Architecture</h3><pre>Azure SQL Database<br>        │<br>        ├── Read lastload.json<br>        ├── Read latest booking timestamp<br>        │<br>        ▼<br>Copy only new records<br>        │<br>        ▼<br>Write Parquet to ADLS<br>        │<br>        ▼<br>Update lastload.json</pre><h3>Technologies Used</h3><ul><li>Azure Data Factory</li><li>Azure SQL Database</li><li>Azure Data Lake Storage Gen2</li><li>Lookup Activity</li><li>Copy Activity</li><li>JSON Dataset</li><li>Parquet Dataset</li></ul><h3>Pipeline Flow</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3iEdaIubnobBXmd0w3XnKA.png" /><figcaption>Incremental Loading using JSON</figcaption></figure><h3>Understanding Each Activity</h3><h3>1. LastLoad Lookup</h3><p>The pipeline first reads the JSON watermark.</p><p>Think of it as opening a bookmark in a book.</p><p>Instead of starting from page 1 every time, you continue from the last<br>page you read.</p><h3>2. LatestLoad Lookup</h3><p>Next, the pipeline asks SQL Server:</p><pre>SELECT MAX(booking_date) AS LatestLoad<br>FROM dbo.FactBookings;</pre><p>This tells us the newest record currently available.</p><h3>3. Copy SQL Data</h3><p>Now we know:</p><p>Old Watermark ✅</p><p>New Watermark ✅</p><p>So we copy only records between them.</p><pre>SELECT *<br>FROM dbo.FactBookings<br>WHERE booking_date &gt; LastLoad<br>AND booking_date &lt;= LatestLoad;</pre><p>Using &gt; avoids copying the same record twice.</p><p>Using &lt;= ensures the latest record is included.</p><h3>4. Update Watermark</h3><p>After the copy succeeds, we overwrite <strong>lastload.json</strong> with the latest<br>timestamp.</p><p>This prepares the pipeline for the next execution.</p><blockquote><em>Notice the order.</em></blockquote><blockquote><em>We update the watermark </em><strong><em>after</em></strong><em> the copy succeeds.</em></blockquote><blockquote><em>If the copy fails, we don’t move the watermark forward.</em></blockquote><p>This prevents data loss.</p><h3>Runtime Execution</h3><pre>Pipeline Starts<br>      │<br>Read JSON Watermark<br>      │<br>Read Latest Timestamp<br>      │<br>Generate Dynamic SQL<br>      │<br>Copy New Records<br>      │<br>Write Parquet<br>      │<br>Update JSON Watermark<br>      │<br>Pipeline Completed</pre><h3>Why is this approach good?</h3><p>✅ Very easy to understand</p><p>✅ Perfect for learning Incremental Loading</p><p>✅ No extra database objects</p><p>✅ Great for Proof of Concepts</p><h3>Limitations</h3><p>As projects grow, this approach becomes harder to manage.</p><p>For example:</p><ul><li>One JSON file per table</li><li>No audit history</li><li>Difficult to manage hundreds of tables</li></ul><p>That’s why enterprise projects usually move to a Watermark Table.</p><p>We’ll build that in <a href="https://medium.com/@jc.prabhaswork/azure-data-factory-incremental-data-loading-series-98fa8399da19"><em>Part 2</em></a>.</p><h3>Common Mistakes</h3><p>❌ Using &gt;= instead of &gt;</p><p>❌ Updating the watermark before the copy completes</p><p>❌ Hardcoding timestamps</p><p>❌ Ignoring time zones</p><h3>🔗 GitHub Repository</h3><p>For the <strong>complete templates, ADF pipeline configurations, expressions, and source code</strong>, please refer to the GitHub repository below:</p><p>👉 <a href="https://github.com/Prabhas92/ADFProject"><strong>View the ADF Project on GitHub</strong></a></p><p>Feel free to explore, reuse, and customize the templates for your own Azure Data Factory projects.</p><p>Happy Learning! 🚀</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d23921dfff28" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>