<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jinson Jacob on Medium]]></title>
        <description><![CDATA[Stories by Jinson Jacob on Medium]]></description>
        <link>https://medium.com/@jinsonj?source=rss-e337722cd6c1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*LFkhRGlvuSjkOT2Jfr4-gw.jpeg</url>
            <title>Stories by Jinson Jacob on Medium</title>
            <link>https://medium.com/@jinsonj?source=rss-e337722cd6c1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 06:59:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jinsonj/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Understanding Precision, Recall, and F1 in RAG Through Scenarios]]></title>
            <link>https://jinsonj.medium.com/understanding-precision-recall-and-f1-in-rag-through-scenarios-de3cd5b65d10?source=rss-e337722cd6c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/de3cd5b65d10</guid>
            <category><![CDATA[agentic-rag]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Jinson Jacob]]></dc:creator>
            <pubDate>Tue, 02 Sep 2025 17:09:48 GMT</pubDate>
            <atom:updated>2025-09-02T17:15:39.262Z</atom:updated>
            <content:encoded><![CDATA[<p>When building a <strong>Retrieval-Augmented Generation (RAG)</strong> system, evaluating your retriever is just as important as evaluating the generator. Metrics like <strong>Precision, Recall, and F1 score</strong> help us understand how well the retriever balances between <strong>finding all relevant documents</strong> and <strong>avoiding noise</strong>.</p><p>To make this concrete, let’s walk through an experiment with the question:</p><p>👉 <em>“Who wrote Hamlet?”</em></p><p>We’ll assume the knowledge base contains <strong>4 passages about Shakespeare</strong> (the relevant docs). Then we’ll test <strong>8 retrieval scenarios</strong> where the system returns different mixes of relevant and irrelevant documents.</p><h3>F1 Score Analogy: The Detective Story</h3><p>Imagine you are a <strong>detective</strong> trying to solve a case:</p><ul><li><strong>Precision</strong> = Of all the clues you collected, how many are actually useful?</li><li>If you pick up lots of random junk (irrelevant clues), precision drops.</li><li><strong>Recall</strong> = Out of all the useful clues at the crime scene, how many did you manage to collect?</li><li>If you miss important fingerprints or CCTV footage, recall drops.</li></ul><p>A good detective — like a good retriever in RAG — must find <strong>enough relevant clues without picking up too much noise</strong>. F1 captures that balance in a single number.</p><h3>Key Intuition</h3><ul><li><strong>Recall only cares about whether you missed any relevant docs.</strong></li><li>If you get <em>all</em> the relevant ones, recall is perfect, no matter how much noise you add.</li><li><strong>Precision, on the other hand, punishes you for bringing irrelevant docs.</strong></li></ul><p>So:</p><ul><li>Adding more <strong>relevant docs → boosts recall (reduces FN)</strong>.</li><li>Adding more <strong>irrelevant docs → lowers precision (increases FP)</strong>.</li></ul><p>Where:</p><ul><li>TP = True Positives (relevant docs retrieved)</li><li>FP = False Positives (irrelevant docs retrieved)</li><li>FN = False Negatives (missed relevant docs)</li></ul><figure><img alt="scenario 9" src="https://cdn-images-1.medium.com/max/1024/1*puAcF8KZNQGU7etfJBBSIQ.png" /><figcaption><strong>S = Shakespeare (relevant documents) , O = Other (irrelevant documents)</strong></figcaption></figure><ul><li>Example: <em>“4S, 2O” means the retriever returned 4 relevant Shakespeare passages and 2 irrelevant ones.</em></li></ul><h3>Visualizing Precision vs Recall</h3><p>To make the trade-offs clearer, here’s a chart plotting Precision vs Recall across all 8 scenarios:</p><ul><li><strong>Top-right corner (1,1):</strong> Perfect retriever (Scenario 1).</li><li><strong>Right edge:</strong> Scenarios 2–5, where recall is perfect but precision falls as noise increases.</li><li><strong>Middle zone:</strong> Scenarios 6–7, where both recall and precision are weak.</li><li><strong>Bottom-left corner (0,0):</strong> Worst case (Scenario 8).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TvKethUOxsrW6pPVmlcWkw.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=de3cd5b65d10" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering Message Queues: Patterns and Benefits]]></title>
            <link>https://jinsonj.medium.com/mastering-message-queues-patterns-and-benefits-740f15a6f8ce?source=rss-e337722cd6c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/740f15a6f8ce</guid>
            <category><![CDATA[message-queue]]></category>
            <category><![CDATA[messaging]]></category>
            <dc:creator><![CDATA[Jinson Jacob]]></dc:creator>
            <pubDate>Fri, 10 Jan 2025 03:42:35 GMT</pubDate>
            <atom:updated>2025-01-10T03:49:16.275Z</atom:updated>
            <content:encoded><![CDATA[<h3>What is a Message Queue?</h3><p>A <strong>message queue</strong> is a form of asynchronous service-to-service communication where producers send messages to a queue, and consumers process them. It acts as a buffer, decoupling the producer and consumer so they can operate independently without being directly connected.</p><p><strong>Key Benefits of Message Queues</strong>:</p><ul><li><strong>Decoupling</strong>: Reduces dependencies between components.</li><li><strong>Scalability</strong>: Allows systems to scale by adding more consumers.</li><li><strong>Fault Tolerance</strong>: Messages are persisted, ensuring reliability even if components fail.</li><li><strong>Asynchronous Processing</strong>: Producers don’t need to wait for consumers to process a message.</li><li><strong>Resiliency</strong>: Ensures smooth communication even during spikes in load.</li></ul><p>Some real-world scenarios where async communication shines are as follows (copied from :<a href="https://bytebytego.com/">https://bytebytego.com/</a>)</p><ul><li>An online store where an order placement triggers real-time calls to inventory, payment, and shipping services. If any of these services experience latency or downtime, the order process stalls, leading to poor user experience and lost revenue. Using a message queue, the order service can immediately enqueue messages for inventory, payment, and shipping.</li><li>IoT systems like smart home devices often involve thousands of sensors sending data to central servers. A synchronous approach can overwhelm the server during peak activity, leading to data loss or delayed responses. Message queues allow sensors to send data without waiting for processing.</li><li>In a microservices architecture, tightly coupled services communicating synchronously can create cascading failures. With message queues, services communicate indirectly, reducing dependency and allowing independent scaling.</li></ul><h3>Message Queue Patterns</h3><h4>1. Request-Reply Pattern</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/590/1*xvGOVSyBdXrAMH7BrswG4A.png" /></figure><p>The <strong>Request-Reply Pattern</strong> is used when a producer expects a response for its message. This pattern ensures two-way communication between the sender and receiver.</p><p><strong>How it Works</strong>:</p><ul><li>The producer sends a <strong>Request</strong> message to a <strong>Request Queue</strong>.</li><li>The consumer processes the message and sends a <strong>Reply</strong> back to a <strong>Response Queue</strong>.</li><li>The producer retrieves the reply from the response queue.</li></ul><p><strong>Use Case</strong>: Payment systems or any scenario where a response (e.g., confirmation or data) is mandatory.</p><p><strong>Advantages</strong>:</p><ul><li>Simplifies request-response communication in distributed systems.</li><li>Ensures a producer can operate asynchronously while waiting for a reply.</li></ul><h4>2. Work Queue Pattern</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/1*KJd9lxhdcCnOjYquKmFhyw.png" /></figure><p>The <strong>Work Queue Pattern</strong> is designed for parallel task distribution among multiple workers. It ensures efficient load balancing, especially for resource-intensive tasks.</p><p><strong>How it Works</strong>:</p><ul><li>Producers push tasks into the queue.</li><li>Workers (e.g., Worker A, Worker B, Worker C) consume tasks from the queue and process them.</li><li>If one worker fails, the task can be picked up by another worker.</li></ul><p><strong>Use Case</strong>: Background jobs like sending bulk emails, video rendering, or processing analytics.</p><p><strong>Advantages</strong>:</p><ul><li>Improves fault tolerance as tasks remain in the queue until processed.</li><li>Scales easily by adding more workers to handle larger workloads.</li></ul><h4>3. Publish-Subscribe Pattern</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*0LC8YYZSJMl8ZLlnxI1hew.png" /></figure><p>The <strong>Publish-Subscribe Pattern</strong> enables broadcasting messages to multiple subscribers simultaneously. Here, the producer doesn’t need to know the subscribers — it simply publishes to a <strong>Topic</strong>.</p><p><strong>How it Works</strong>:</p><ul><li>A producer (e.g., Order Service) publishes an event (e.g., OrderCreated) to a topic.</li><li>Multiple subscribers (e.g., Inventory Service, Shipping Service, Notification Service) listen to the topic and act on the event.</li></ul><p><strong>Use Case</strong>: E-commerce platforms where an order event triggers multiple actions: updating inventory, preparing shipments, and sending notifications.</p><p><strong>Advantages</strong>:</p><ul><li>Perfect for event-driven architectures.</li><li>Decouples services, allowing independent development and scaling of subscribers.</li></ul><h3>Conclusion</h3><p>Message queues are the backbone of scalable, decoupled, and resilient systems. By implementing patterns like <strong>Request-Reply</strong>, <strong>Work Queue</strong>, and <strong>Publish-Subscribe</strong>, developers can design robust architectures tailored to their application’s needs. Whether you’re handling asynchronous tasks or enabling real-time event-driven communication, message queues provide the flexibility and reliability required in modern distributed systems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=740f15a6f8ce" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Mesh with Data Federation vs. Hybrid Data Mesh]]></title>
            <link>https://jinsonj.medium.com/data-mesh-with-data-federation-vs-hybrid-data-mesh-8f1ddb75f965?source=rss-e337722cd6c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/8f1ddb75f965</guid>
            <category><![CDATA[data-mesh-architecture]]></category>
            <category><![CDATA[data-mesh]]></category>
            <dc:creator><![CDATA[Jinson Jacob]]></dc:creator>
            <pubDate>Sun, 16 Jun 2024 17:30:21 GMT</pubDate>
            <atom:updated>2024-06-16T17:30:21.530Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Data Mesh with Data Federation</strong> and <strong>Hybrid Data Mesh</strong> are both approaches to data architecture, but they have different focuses and implementations. Here’s a comparison to highlight their differences and help determine which might be best for your needs:</p><h3>Data Mesh with Data Federation : <a href="https://medium.com/@jinsonj/data-mesh-with-data-federation-4084fe29f245">see here</a></h3><h3>Hybrid Data Mesh : <a href="https://jinsonj.medium.com/hybrid-data-mesh-320686b6e424">see here</a></h3><h3>Data Mesh with Data Federation</h3><h4>Characteristics:</h4><p>1. <strong>Decentralized Data Ownership</strong>:</p><ul><li>Data is owned and managed by individual domains.</li><li>Each domain is responsible for its own data products, treating data as a product.</li></ul><p>2. <strong>Federated Access</strong>:</p><ul><li>Uses data federation techniques to integrate and query data across multiple domains without physically moving the data.</li><li>A federated query engine provides a virtual layer to access data from various sources in real-time.</li></ul><p>3. <strong>Seamless Integration</strong>:</p><ul><li>Enables real-time or near-real-time querying and analysis of data across different domains.</li><li>Reduces data duplication and ensures data consistency.</li></ul><h4>Advantages:</h4><ul><li><strong>Decentralized Management</strong>: Empowers domain teams to manage their own data, ensuring data quality and relevance.</li><li><strong>Real-Time Access</strong>: Provides real-time or near-real-time access to data across domains.</li><li><strong>Scalability</strong>: Easily integrates new data sources without the need for significant rearchitecting.</li></ul><h4>Use Case:</h4><ul><li>Suitable for organizations with well-defined domain boundaries and the need for real-time data integration and access.</li></ul><h3>Hybrid Data Mesh</h3><h4>Characteristics:</h4><p>1. <strong>Centralized and Decentralized Components</strong>:</p><ul><li>Combines decentralized data ownership with centralized data storage for certain datasets.</li><li>Data from various domains is ingested into a central data lake or data warehouse for easier management and processing.</li></ul><p>2. <strong>Batch and Real-Time Processing</strong>:</p><ul><li>Allows both batch processing (centralized data lake/warehouse) and real-time processing (decentralized data management).</li></ul><p>3. <strong>Efficiency in Large-Scale Data Processing</strong>:</p><ul><li>Centralized storage can handle large-scale data processing and analytics more efficiently.</li></ul><h4>Advantages:</h4><ul><li><strong>Centralized Analytics</strong>: Facilitates centralized analytics and reporting, leveraging the scalability of data lakes/warehouses.</li><li><strong>Simplified Management</strong>: Central repository simplifies data management and governance for large datasets.</li><li><strong>Flexibility</strong>: Provides flexibility in data processing, supporting both batch and real-time use cases.</li></ul><h4>Use Case:</h4><ul><li>Suitable for organizations with a mix of real-time and batch processing needs, and where centralized storage can enhance efficiency and scalability.</li></ul><h3>Which is Best: Hybrid Data Mesh or Data Mesh with Data Federation?</h3><p>The best approach depends on your organization’s specific needs, data architecture, and operational requirements.</p><h4>Choose Data Mesh with Data Federation if:</h4><ul><li>You prioritize decentralized data ownership and governance.</li><li>Real-time or near-real-time data access and integration are crucial for your operations.</li><li>You want to minimize data duplication and ensure data consistency across domains.</li></ul><h4>Choose Hybrid Data Mesh if:</h4><ul><li>You need a balance between decentralized data ownership and centralized data processing.</li><li>You have significant batch processing requirements that can benefit from centralized data storage.</li><li>Your organization requires a scalable solution for large-scale data analytics and reporting.</li></ul><h3>Conclusion</h3><p>Both approaches offer unique benefits, and the choice between them should be guided by your organization’s data strategy, infrastructure, and operational goals. In some cases, a hybrid approach that combines elements of both Data Mesh with Data Federation and Hybrid Data Mesh might be the most effective solution, providing the advantages of decentralized ownership with the efficiencies of centralized processing.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f1ddb75f965" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hybrid Data Mesh]]></title>
            <link>https://jinsonj.medium.com/hybrid-data-mesh-320686b6e424?source=rss-e337722cd6c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/320686b6e424</guid>
            <category><![CDATA[data-mesh-architecture]]></category>
            <category><![CDATA[data-mesh]]></category>
            <dc:creator><![CDATA[Jinson Jacob]]></dc:creator>
            <pubDate>Sun, 16 Jun 2024 17:27:12 GMT</pubDate>
            <atom:updated>2024-06-16T17:27:12.296Z</atom:updated>
            <content:encoded><![CDATA[<h4>Hybrid Data Mesh : A Hybrid Data Mesh combines the decentralized principles of a data mesh with the practicalities of centralized data storage. This approach allows domains to manage and govern their data while still leveraging a central repository for efficiency in data storage, processing, and analytics.</h4><h3><strong>Step 1: Define the Domains and Data Products</strong></h3><p><strong>Domains:</strong></p><ol><li><strong>Sales Domain</strong>: Manages sales data.</li><li><strong>Customer Domain</strong>: Manages customer data.</li><li><strong>Inventory Domain</strong>: Manages inventory data.</li></ol><p>Each domain will continue to own and manage its data independently, treating it as a product. Additionally, there will be a centralized data lake or data warehouse to aggregate and process data from all domains for large-scale analytics.</p><p><strong>Step 2: Data Products and APIs</strong></p><ol><li><strong>Sales Data Product</strong>:</li></ol><ul><li><strong>API Endpoint</strong>: <a href="https://api.salesdomain.com/sales">https://api.salesdomain.com/sales</a></li></ul><p><strong>Customer Data Product</strong>:</p><ul><li><strong>API Endpoint</strong>: <a href="https://api.customerdomain.com/customers">https://api.customerdomain.com/customers</a></li></ul><p><strong>Inventory Data Product</strong>:</p><ul><li><strong>API Endpoint</strong>: <a href="https://api.inventorydomain.com/inventory">https://api.inventorydomain.com/inventory</a></li></ul><h3>Step 3: Centralized Data Lake/Warehouse</h3><p>Set up a centralized data lake (e.g., AWS S3) or data warehouse (e.g., Amazon Redshift) to store and process aggregated data from all domains.</p><h4>ETL/ELT Pipelines:</h4><p>1. <strong>Sales Data Ingestion</strong>:</p><ul><li>Use an ETL tool (e.g., AWS Glue, Apache NiFi) to extract data from the SQL Server database.</li><li>Transform the data as needed.</li><li>Load the transformed data into the centralized data lake/warehouse.</li></ul><p>2. <strong>Customer Data Ingestion</strong>:</p><ul><li>Use an ETL tool to extract data from Salesforce.</li><li>Transform the data as needed.</li><li>Load the transformed data into the centralized data lake/warehouse.</li></ul><p>3. <strong>Inventory Data Ingestion</strong>:</p><ul><li>Use an ETL tool to extract data from the Oracle system.</li><li>Transform the data as needed.</li><li>Load the transformed data into the centralized data lake/warehouse.</li></ul><h3>Step 4: Federated Query Engine</h3><p>Deploy a federated query engine (e.g., Presto, Trino) to enable real-time querying across the decentralized data sources and the centralized data repository.</p><ul><li>Choose a federated query engine like <strong>Presto</strong>, <strong>Trino</strong>, or <strong>Denodo</strong>.</li><li>Install and configure the query engine on a server or use a managed service.</li></ul><h4>2. Configure Connectors for Decentralized Data Sources</h4><p>Configure the federated query engine to connect to each decentralized domain-specific data source.</p><p>Sales Data (SQL Server)</p><pre>connector.name=sqlserver</pre><pre>connection-url=jdbc:sqlserver://salesdb:1433;databaseName=salesdb</pre><pre>connection-user=sales_user</pre><pre>connection-password=sales_password</pre><p>Customer Data (Salesforce)</p><pre>connector.name=salesforce</pre><pre>username=your_salesforce_username</pre><pre>password=your_salesforce_password</pre><p>Inventory Data (Oracle)</p><pre>connector.name=oracle</pre><pre>connection-url=jdbc:oracle:thin:@inventorydb:1521:orcl</pre><pre>connection-user=inventory_user</pre><pre>connection-password=inventory_password</pre><h4>3. Configure Connectors for the Centralized Data Repository</h4><p>Configure the federated query engine to connect to the centralized data lake or data warehouse.</p><p>Centralized Data Lake (AWS S3)</p><pre>connector.name=hive</pre><pre>hive.metastore.uri=thrift://metastore:9083</pre><pre>hive.s3.aws-access-key=your_aws_access_key</pre><pre>hive.s3.aws-secret-key=your_aws_secret_key</pre><pre>hive.s3.bucket-name=your_s3_bucket_name</pre><p>Centralized Data Warehouse (Amazon Redshift)</p><pre>connector.name=redshift</pre><pre>connection-url=jdbc:redshift://redshift-cluster:5439/yourdatabase</pre><pre>connection-user=redshift_user</pre><pre>connection-password=redshift_password</pre><h3>Scenarios and Example Queries</h3><h4>Scenario 1: Real-Time Data Access</h4><p>You need to perform real-time analytics by combining live data from the decentralized systems (Sales, Customer, and Inventory).</p><p>Example Federated Query</p><pre>SELECT</pre><pre>s.order_id,</pre><pre>s.order_date,</pre><pre>c.customer_name,</pre><pre>i.product_name,</pre><pre>s.quantity,</pre><pre>s.total_amount,</pre><pre>i.stock_level</pre><pre>FROM</pre><pre>salesdb.sales s</pre><pre>JOIN</pre><pre>salesforce.customers c ON s.customer_id = c.customer_id</pre><pre>JOIN</pre><pre>oracledb.inventory i ON s.product_id = i.product_id</pre><pre>WHERE</pre><pre>s.order_date &gt;= &#39;2024-01-01&#39;;</pre><h4>Scenario 2: Large-Scale Batch Processing</h4><p>You need to generate a monthly sales report by aggregating historical data stored in the centralized data warehouse.</p><p><strong>Example Centralized Query</strong></p><pre>SELECT</pre><pre>order_date,</pre><pre>SUM(total_amount) as total_sales</pre><pre>FROM</pre><pre>redshift.sales</pre><pre>GROUP BY</pre><pre>order_date;</pre><h3>Benefits of This Approach</h3><h4>Real-Time and Batch Processing</h4><ul><li><strong>Real-Time Processing</strong>: The federated query engine allows real-time access and querying across decentralized systems without moving the data.</li><li><strong>Batch Processing</strong>: Centralized data storage supports large-scale batch processing and analytics, leveraging the scalability of data lakes and warehouses.</li></ul><h4>Unified Data Access</h4><ul><li>Provides a unified interface for querying both decentralized and centralized data sources.</li><li>Simplifies data integration and analysis across the entire data landscape.</li></ul><h4>Flexibility and Scalability</h4><ul><li>Easily integrates new data sources by adding new connectors to the federated query engine.</li><li>Scales with the growth of the organization’s data needs, supporting both decentralized and centralized data management strategies.</li></ul><h3>Conclusion</h3><p>By configuring a federated query engine to connect to both decentralized domain-specific data sources and the centralized data repository, you can achieve a Hybrid Data Mesh architecture that supports seamless data integration, real-time analytics, and large-scale batch processing. This approach provides the flexibility and scalability needed to manage and analyze data effectively across the entire organization.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=320686b6e424" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Mesh with Data Federation]]></title>
            <link>https://jinsonj.medium.com/data-mesh-with-data-federation-4084fe29f245?source=rss-e337722cd6c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/4084fe29f245</guid>
            <category><![CDATA[data-federation]]></category>
            <category><![CDATA[data-mesh]]></category>
            <category><![CDATA[data-mesh-architecture]]></category>
            <dc:creator><![CDATA[Jinson Jacob]]></dc:creator>
            <pubDate>Sun, 16 Jun 2024 17:00:13 GMT</pubDate>
            <atom:updated>2024-06-16T17:00:13.377Z</atom:updated>
            <content:encoded><![CDATA[<p>Integrating and managing data across multiple domains using both Data Mesh principles and Data Federation techniques</p><p><strong>Data Mesh</strong> is a decentralized approach to data architecture that treats data as a product and assigns ownership of the data to the domain teams that generate it. It shifts the focus from centralized data management to distributed data management, empowering domain teams to manage, serve, and consume data independently while adhering to common governance standards.</p><p><strong>Data Federation</strong> is a method of data management that allows an organization to integrate data from multiple sources without physically moving or copying the data to a central repository. Instead, it provides a unified view of the data through a virtual layer, enabling users to query and analyze data across disparate systems as if it were a single dataset.</p><p>Creating a Data Mesh and Data Federation for Sales, Customer, and Inventory Data</p><p>Imagine a retail company that has multiple data sources:</p><ul><li><strong>Sales Data</strong>: Stored in a transactional database (e.g., SQL Server).</li><li><strong>Customer Data</strong>: Stored in a CRM system (e.g., Salesforce).</li><li><strong>Inventory Data</strong>: Stored in a warehouse management system (e.g., Oracle).</li></ul><p>The company wants to analyze sales performance, customer behavior, and inventory levels in a unified manner without moving all data into a single data warehouse.</p><h3>Step 1: Define the Domains and Data Products</h3><h4>Domains:</h4><ol><li><strong>Sales Domain</strong>: Manages sales data stored in a SQL Server database.</li><li><strong>Customer Domain</strong>: Manages customer data stored in a Salesforce CRM.</li><li><strong>Inventory Domain</strong>: Manages inventory data stored in an Oracle warehouse management system.</li></ol><h4>Data Products:</h4><p>Each domain will expose its data as a product with well-defined APIs and schemas.</p><p>1. <strong>Sales Data Product</strong>:</p><ul><li><strong>API Endpoint</strong>: <a href="https://api.salesdomain.com/sales">https://api.salesdomain.com/sales</a></li></ul><p>2. <strong>Customer Data Product</strong>:</p><ul><li><strong>API Endpoint</strong>: <a href="https://api.customerdomain.com/customers">https://api.customerdomain.com/customers</a></li></ul><p>3. <strong>Inventory Data Product</strong>:</p><ul><li><strong>API Endpoint</strong>: <a href="https://api.inventorydomain.com/inventory">https://api.inventorydomain.com/inventory</a></li></ul><h3>Step 2: Implement Data Federation</h3><p>To enable seamless integration and querying of data across these domains, we will set up a federated query engine.</p><h4>Federated Query Engine Setup:</h4><p>1. <strong>Select a Federated Query Engine</strong>: We will use Presto (also known as Trino) for this purpose.</p><p>2. <strong>Configure Connectors</strong>:</p><p> <strong>Sales Data (SQL Server)</strong>:</p><p>properties</p><pre>connector.name=sqlserver<br>connection-url=jdbc:sqlserver://salesdb:1433;databaseName=salesdb<br>connection-user=sales_user<br>connection-password=sales_password</pre><p> <strong>Customer Data (Salesforce)</strong>:</p><p>properties</p><pre>connector.name=salesforce<br>username=your_salesforce_username<br>password=your_salesforce_password</pre><p> <strong>Inventory Data (Oracle)</strong>:</p><p>properties</p><pre>connector.name=oracle<br>connection-url=jdbc:oracle:thin:@inventorydb:1521:orcl<br>connection-user=inventory_user<br>connection-password=inventory_password</pre><p>3. <strong>Deploy Presto</strong>:</p><ul><li>Install and configure Presto on a server or use a managed service.</li><li>Configure Presto to connect to the above data sources using the connectors.</li></ul><h3>Step 3: Create Federated Queries</h3><p>Now that the federated query engine is set up, you can create queries that join data from the different domains.</p><pre>SELECT<br>s.order_id,<br>s.order_date,<br>c.customer_name,<br>i.product_name,<br>s.quantity,<br>s.total_amount,<br>i.stock_level<br>FROM<br>sales.sales s<br>JOIN<br>customer.customers c ON s.customer_id = c.customer_id<br>JOIN<br>inventory.inventory i ON s.product_id = i.product_id<br>WHERE<br>s.order_date &gt;= &#39;2024–01–01&#39;;</pre><p>By following these steps, you can implement a Data Mesh with Data Federation, enabling seamless and efficient data integration and querying across your Sales, Customer, and Inventory domains. This approach will ensure that data remains decentralized while still being accessible and manageable in a unified manner.</p><p><strong>Conclusion:</strong></p><p>Data Federation is an essential component of a Data Mesh architecture. It provides the necessary infrastructure to integrate, query, and manage data across decentralized domains, ensuring that data is accessible, discoverable, and governed consistently across the organization. This integration supports the core principles of Data Mesh by enabling domains to operate independently while maintaining a cohesive and unified data ecosystem.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4084fe29f245" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding micro-services design patterns]]></title>
            <link>https://jinsonj.medium.com/understanding-micro-services-design-patterns-e2d7ab46ca92?source=rss-e337722cd6c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/e2d7ab46ca92</guid>
            <category><![CDATA[microservices]]></category>
            <dc:creator><![CDATA[Jinson Jacob]]></dc:creator>
            <pubDate>Sun, 21 Apr 2024 19:03:42 GMT</pubDate>
            <atom:updated>2025-09-02T18:03:58.996Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sbzqyco126mQKYcAUwaakQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wxR9cE1HAqm3iQDnQE8okw.png" /></figure><ul><li>In the <strong>first diagram</strong>: It’s a simple monolithic setup (web → app → DB).</li><li>In the <strong>second diagram</strong>: ESB is introduced to mediate between the web tier and backend services, making integration easier and more manageable. <strong>Routes requests from one system to the correct backend service. Transforms data formats</strong> (e.g., JSON ↔ XML ↔ SOAP).</li><li>In the <strong>third diagram</strong>: The architecture evolves further into <strong>microservices</strong> with load balancing, where ESB isn’t needed because communication is handled via API gateways / service meshes.</li><li>Each service could be developed, tested, deployed, and maintained independently. Thus, each smaller (micro) service has its own unique life cycle. Furthermore, since each service is loosely coupled (interacting with other services using HTTP/HTTPS), we can do the following:</li><li>Scale up or scale down (based on the service traffic).</li><li>Address any runtime faults (boot up the service backup).</li><li>Make new changes (change impact is limited to the service).</li></ul><p>Therefore, through the complete realization of decoupled architecture in the micro-services, we address key issues of scalability, fault tolerance, high availability, and flexibility.</p><h3><strong>Core design patterns for microservices architecture</strong></h3><h3>1. Decomposition Design Patterns</h3><p>These patterns deal with <strong>how to break a monolith into microservices</strong>.</p><h3>a. Decomposing by Business Capability</h3><ul><li>Break down services according to business functions (e.g., <em>Payments, Orders, Shipping</em>).</li><li>Each service maps to a core business capability.</li><li>Example: Amazon → “Order Service,” “Payment Service,” “Delivery Service.”</li></ul><h3>b. Decomposing by Domain / Sub-domain</h3><ul><li>Based on <strong>Domain-Driven Design (DDD)</strong>.</li><li>You split services into <strong>bounded contexts</strong> like <em>Customer, Product, Inventory</em>.</li><li>Example: In an e-commerce system → “Product Catalog,” “Cart,” “User Profile.”</li></ul><h3>2. Integration Design Patterns</h3><p>These define <strong>how microservices communicate</strong>.</p><h3>a. API Gateway Pattern</h3><ul><li>A single entry point for clients.</li><li>Routes requests to the right microservice, applies authentication, logging, throttling.</li><li>Example: Netflix Zuul / Spring Cloud Gateway.</li></ul><h3>b. Aggregator Pattern</h3><ul><li>One service calls multiple services, aggregates results, and returns one response.</li><li>Example: A “User Profile API” calls <em>Orders Service</em>, <em>Payment Service</em>, <em>Shipping Service</em> and returns combined profile info.</li></ul><h3>c. Chained Microservices Pattern</h3><ul><li>A request flows through a <strong>chain of microservices</strong>, each enriching data before passing it on.</li><li>Example: An “Order Checkout” request goes → <em>Cart Service</em> → <em>Payment Service</em> → <em>Inventory Service</em>.</li></ul><h3>3. Data Management Patterns</h3><p>How services handle <strong>databases and data consistency</strong>.</p><h3>a. Database per Service (recommended)</h3><ul><li>Each microservice owns its database.</li><li>Ensures loose coupling.</li><li>Example: <em>Order Service</em> has its DB, <em>Payment Service</em> has its DB.</li></ul><h3>b. Shared Database (anti-pattern, but sometimes used)</h3><ul><li>Multiple services share one big DB.</li><li>Easier to start but leads to tight coupling.</li><li>Example: Monolithic DB with many schemas.</li></ul><h3>c. CQRS (Command Query Responsibility Segregation)</h3><ul><li>Split <strong>read and write operations</strong> into separate models/databases</li><li>Example: Writes go to transactional DB; reads served from optimized query DB.</li></ul><h3>4. Cross-Cutting Patterns</h3><p>Concerns that affect <strong>all microservices</strong>.</p><h3>a. Service Discovery Pattern</h3><ul><li>Services find each other dynamically (instead of hardcoding addresses).</li><li>Example: Eureka, Consul, Kubernetes DNS.</li></ul><h3>b. Circuit Breaker Pattern</h3><ul><li>Prevents cascading failures when a service is down.</li><li>If Service B keeps failing, Service A “breaks the circuit” and uses a fallback.</li><li>Example: Netflix Hystrix, Resilience4j.</li></ul><h3>c. Log Aggregation Pattern</h3><ul><li>Collect logs from all microservices into one centralized system.</li><li>Helps debugging and monitoring.</li><li>Example: ELK stack (Elasticsearch, Logstash, Kibana), Splunk.</li></ul><p><strong>In short:</strong></p><ul><li><strong>Decomposition patterns</strong> → how to split the system.</li><li><strong>Integration patterns</strong> → how services talk.</li><li><strong>Data management patterns</strong> → how to handle databases.</li><li><strong>Cross-cutting patterns</strong> → reliability, discovery, and observability</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e2d7ab46ca92" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CAP theorem]]></title>
            <link>https://jinsonj.medium.com/cap-theorem-dec644f7a2b0?source=rss-e337722cd6c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/dec644f7a2b0</guid>
            <category><![CDATA[partition-tolerance]]></category>
            <category><![CDATA[consistency]]></category>
            <category><![CDATA[availability]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Jinson Jacob]]></dc:creator>
            <pubDate>Wed, 27 Dec 2023 18:47:27 GMT</pubDate>
            <atom:updated>2023-12-27T20:55:47.687Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*HD0cOWiMw1dqtrSaTcfHtg.png" /></figure><p>The CAP theorem, also known as Brewer’s theorem, is a concept in distributed computing that describes the <strong>trade-offs and limitations among three key properties </strong>of a distributed system: <strong>Consistency, Availability, and Partition Tolerance</strong>. The theorem was introduced by computer scientist Eric Brewer in 2000.</p><p><strong>The three components of the CAP theorem are:</strong></p><p>1. Consistency [C]: In a distributed system, consistency refers to the idea that all nodes in the system see the<strong> same data at the same time</strong>. In other words, when a write operation is completed, all subsequent read operations should reflect that write</p><p>2. Availability [A]: Availability implies that every request to the system receives a response, <strong>without guaranteeing that it contains the most recent version of the data</strong>. A system is considered available as long as it responds to client requests, even if some nodes are unavailable or experiencing delays.</p><p>3. Partition Tolerance [P]: Partition tolerance means that the <strong>system continues to operate despite network partitions, </strong>or communication failures, that may occur between nodes in the distributed system. Partition tolerance is crucial for maintaining system functionality in the face of network issues.</p><p><strong>Distributed System:</strong></p><p>A distributed system is a system that consists of multiple independent components or nodes that work together to achieve a common goal. These nodes can be geographically dispersed and communicate with each other to share data and coordinate their activities. Distributed systems are designed to improve performance, fault tolerance, and scalability.</p><p>The CAP theorem states that in a distributed system, it is impossible to simultaneously achieve all three of these properties. The theorem introduces a trade-off scenario where a system can prioritize any two of the three properties, but it must sacrifice the third. The three possible combinations are:</p><p>· CA (Consistency and Availability without Partition Tolerance): In this scenario, the system ensures consistency and availability but cannot tolerate network partitions. If a partition occurs, the system may become unavailable or inconsistent.</p><p>· CP (Consistency and Partition Tolerance without Availability): This combination prioritizes consistency and partition tolerance. The system remains consistent even in the presence of network partitions, but availability may be compromised. During a partition, the system may choose to be temporarily unavailable to maintain consistency.</p><p>· AP (Availability and Partition Tolerance without Consistency): This combination emphasizes availability and partition tolerance. The system prioritizes responsiveness and continues to operate even in the presence of network partitions, but it may sacrifice immediate consistency. Different nodes in the system may have different views of the data.</p><p><strong>Here are some characteristics of applications that might prefer a CA model:</strong></p><p>E-commerce platforms often adopt a blend of Consistency and Availability, following a CA model. This means that they prioritize maintaining a consistent and accurate view of product catalogs, prices, and inventory across all nodes in the distributed system.</p><p>In a CA system, temporary unavailability during network partitions may be deemed acceptable. During network disruptions or partitions, where communication between nodes is affected, the system may choose to become temporarily unavailable to avoid conflicting or inconsistent states. This ensures that users do not see outdated or conflicting product information.</p><p><strong>Here are some characteristics of applications that might prefer a CP model:</strong></p><p>Financial systems (banking, stock trading) typically prioritize strong consistency, adhering to a CP (Consistency and Partition Tolerance without Availability) model. Strong consistency ensures that all nodes within the distributed system see the same order of transactions and maintain a consistent state, even in the face of network partitions.</p><p>Strong consistency in financial systems requires strict synchronization of transactions. Every transaction, such as deposits, withdrawals, or stock trades, must be processed consistently across all nodes to avoid discrepancies in account balances or financial records.</p><p><strong>Here are some characteristics of applications that might prefer a AP model:</strong></p><p>Social media platforms often prioritize delivering updates to users in real-time. An AP model allows the platform to remain available and responsive during network partitions, allowing users to post and view content continuously. Temporary inconsistencies in the visibility of posts across different nodes may be acceptable in exchange for continuous availability.</p><p>CDNs prioritize serving content to users with low latency and high availability. In scenarios where nodes in the CDN are geographically distributed and may experience network partitions, an AP model allows the system to remain available and responsive to user requests, even if it means that different nodes may temporarily serve slightly outdated content.</p><p>Applications that support real-time collaborative editing, such as collaborative documents or code editors, may prefer an AP model. Users need to see immediate updates made by collaborators, and maintaining high availability is crucial. While eventual consistency is accepted, the system aims to provide a responsive and real-time collaborative experience.</p><p>IoT applications often involve a large number of devices that communicate over diverse and sometimes unreliable networks. In an AP model, IoT systems prioritize availability and responsiveness, allowing devices to continue to function and communicate even in the presence of network partitions. Eventual consistency is accepted for data recorded by different devices.</p><p>Online gaming platforms prioritize low-latency interactions and continuous gameplay. An AP model allows these platforms to remain available, allowing players to interact and play in real-time. Temporary inconsistencies in player states across different gaming servers may be acceptable, as long as the gaming experience remains responsive.</p><p>Ephemeral messaging applications, where messages have a short lifespan, may opt for an AP model. The priority is on delivering messages quickly to users, and availability is crucial. While messages may eventually be consistent across devices, immediate availability takes precedence.</p><p>Search engines aim to provide fast and relevant search results. In an AP model, search engines prioritize availability and responsiveness to deliver search results quickly. Temporary inconsistencies in the indexing or ranking of content across different nodes may be acceptable for the sake of continuous service.</p><p>Caching systems, where data is cached across distributed nodes, may opt for an AP model. The emphasis is on serving cached data quickly and efficiently to users, and temporary inconsistencies in the cache across different nodes may be acceptable.</p><h3><strong>Handling Tradeoffs</strong></h3><p>As discussed above, CAP theorem brings tradeoffs, however, there are strategies that can be employed to handle the trade-offs imposed by the CAP theorem.</p><p><strong>Eventual Consistency:</strong></p><p>Eventual consistency is a key characteristic of the AP model. It means that, over time, all replicas or nodes in the distributed system will converge to a consistent state. However, during periods of network partitions or delays in communication, different nodes may have temporary inconsistencies. <strong>AP systems make a trade-off by prioritizing availability and partition tolerance over strict consistency.</strong></p><p><strong>Quorum Systems:</strong></p><p>In distributed databases, quorum systems are often used for read and write operations. For example, a database might require a quorum of nodes to acknowledge a write operation as successful or to retrieve data in a read operation. This approach ensures that data consistency is maintained even in the presence of node failures.</p><p><strong>Quorum systems help handle network partitions</strong> by ensuring that nodes on both sides of a partition do not independently make conflicting decisions. The requirement for a quorum ensures that only a majority of nodes can make progress, preventing the system from diverging into inconsistent states.</p><p>In a system with five nodes, a common quorum might require three nodes to agree. This is often referred to as a “majority quorum.” Other quorum configurations are possible, such as requiring a strict majority or a fixed number of nodes.</p><p>Quorum systems are often associated with consensus algorithms, which are protocols used to achieve agreement among a set of nodes in a distributed system. Well-known consensus algorithms, such as Paxos and Raft, use a quorum-based approach to ensure that a majority of nodes agree on a value or decision.</p><p><strong>Multi-Datacenter Replication:</strong></p><p>Replicating data across multiple data centers enhances fault tolerance and high availability. If one data center experiences a failure, the application can continue to operate using data from other geographically distributed data centers.</p><p>Multi-datacenter replication involves decisions about the consistency model to be employed. Different systems may choose varying levels of consistency, such as eventual consistency, strong consistency, or causal consistency, depending on the application requirements and trade-offs.</p><p>Replication can occur synchronously or asynchronously. Synchronous replication ensures that data is replicated to all data centers before acknowledging a write operation, while asynchronous replication allows for a delay between the write operation and replication.</p><p>The topology of the network connecting data centers is a critical consideration. Choices include star topologies, where all data centers connect to a central hub, or mesh topologies, where data centers connect to multiple other data centers.</p><p>Global load balancing is often used in conjunction with multi-datacenter replication to route user requests to the nearest or most responsive data center. This helps optimize the user experience and distribute the load efficiently.</p><p>Popular databases and distributed systems, such as Apache Cassandra, MongoDB, and relational databases with replication features, often provide mechanisms for multi-datacenter replication to support the aforementioned topics.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dec644f7a2b0" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>