<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sureshkumar Mani on Medium]]></title>
        <description><![CDATA[Stories by Sureshkumar Mani on Medium]]></description>
        <link>https://medium.com/@surezms?source=rss-630dfd05eee0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*DdqM0TE_kjb5itdRilCYlg.jpeg</url>
            <title>Stories by Sureshkumar Mani on Medium</title>
            <link>https://medium.com/@surezms?source=rss-630dfd05eee0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 16:22:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@surezms/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Knowldge Graph]]></title>
            <link>https://medium.com/@surezms/knowldge-graph-a182b9aca8a2?source=rss-630dfd05eee0------2</link>
            <guid isPermaLink="false">https://medium.com/p/a182b9aca8a2</guid>
            <dc:creator><![CDATA[Sureshkumar Mani]]></dc:creator>
            <pubDate>Fri, 26 Sep 2025 18:36:45 GMT</pubDate>
            <atom:updated>2025-09-27T02:18:29.079Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Knowledge Graphs Explained: Simple Guide with Examples and Enterprise Applications</strong></p><p>Knowledge graphs are a way of organizing and storing information that mirrors how we naturally think about relationships in the real world. Think of them as smart networks that connect different pieces of information together.</p><p><strong>What is a Knowledge Graph? A Simple Example</strong></p><p>Imagine you’re organizing information about your company. Instead of storing data in separate tables like traditional databases, a knowledge graph connects everything together like a web of relationships.</p><p><strong>Simple Example:</strong></p><ul><li><strong>Entities (Nodes)</strong>: John, Microsoft, Engineering Department, AI Project</li><li><strong>Relationships (Edges)</strong>:</li><li>John <strong>works for</strong> Microsoft</li><li>John <strong>belongs to</strong> Engineering Department</li><li>John <strong>leads</strong> AI Project</li><li>AI Project <strong>is funded by</strong> Microsoft</li></ul><p>This creates a connected network where you can easily trace relationships: “Who leads projects funded by Microsoft in the Engineering Department?” The answer flows naturally through the connections.</p><p><strong>Real Company Example</strong></p><p># Simple knowledge graph representation<br> Company Knowledge Graph:<br> — Employee: “Sarah Johnson” <br> — works_for → “Standard Chartered Bank”<br> — member_of → “Data Analytics Team”<br> — reports_to → “Michael Chen” <br> — works_on → “Customer Segmentation Project”<br> — has_skill → “Python, Tableau”</p><p><strong>Knowledge Graphs vs Vector Databases: Key Differences</strong></p><p><strong>Vector Databases</strong></p><p>Vector databases store information as numerical representations (embeddings) in high-dimensional space. They’re excellent for finding similar content.</p><p><strong>How they work:</strong></p><ul><li>Convert text “John works at Microsoft” into numbers like [0.2, 0.8, 0.1, 0.9…]</li><li>Find similar content by calculating mathematical distances between vectors[6]</li><li>Good for: “Find documents similar to this query”</li></ul><p><strong>Knowledge Graphs Benefits</strong></p><p>Knowledge graphs store explicit relationships between entities that you can read and understand.</p><p><strong>How they work:</strong></p><ul><li>Store clear connections: John → WORKS_AT → Microsoft</li><li>Navigate relationships directly: “Who works at Microsoft?”</li><li>Good for: Complex questions requiring multi-step reasoning</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UBK5y2O2TlQZDDR3dH-qag.png" /></figure><p><strong>Enterprise Knowledge Graph: Pros and Cons</strong></p><p><strong>1. Explainable Decisions</strong><br> Unlike vector databases that work like black boxes, knowledge graphs show exactly how they reached a conclusion. Critical for compliance and auditing in regulated industries.</p><p><strong>2. Complete and Accurate Answers</strong><br> When you ask “List all employees in the Engineering department,” a knowledge graph returns exactly that — no more, no less. Vector databases might return incomplete lists or include unrelated results.</p><p><strong>3. Multi-Source Data Integration</strong><br> Knowledge graphs excel at connecting data from different enterprise systems — HR databases, project management tools, financial systems — into one unified view.</p><p><strong>4. Dynamic Updates</strong><br> Add new employees, departments, or projects without rebuilding the entire system. Simply add new nodes and relationships.</p><p><strong>5. Better Context for AI</strong><br> Provides structured context that improves AI applications, reducing hallucinations and improving accuracy.</p><p>Challenges in Integration:</p><p><strong>1. Initial Complexity</strong><br> Requires upfront design of schemas and ontologies before becoming useful. Need to define what entities and relationships matter for your business.</p><p><strong>2. Integration Challenges</strong><br> Enterprise systems are often siloed with messy, inconsistent data. Connecting multiple APIs and ensuring data quality requires significant effort.</p><p><strong>3. Specialized Knowledge Required</strong><br> Teams may need to learn graph query languages like SPARQL or Cypher. Different from traditional SQL database skills.</p><p><strong>Enterprise Use Cases</strong></p><p><strong>Financial Services</strong></p><p><strong>Example</strong>: A bank uses knowledge graphs to model customer relationships, account hierarchies, and transaction patterns. This enables sophisticated fraud detection by tracing unusual connection patterns and ensures compliance by maintaining clear audit trails.</p><p><strong>Human Resources</strong></p><p><strong>Example</strong>: Connect employee skills, project requirements, team structures, and availability. Enables intelligent resource allocation: “Find senior developers with React experience who aren’t on critical projects”.</p><p><strong>Customer Service</strong></p><p><strong>Example</strong>: Link customer profiles, purchase history, support tickets, and product information. Support agents get complete context: “Show me this customer’s recent purchases, related warranty issues, and previous support interactions”.</p><p><strong>Implementation Example</strong></p><p># Enterprise API to Knowledge Graph<br> from neo4j import GraphDatabase<br> <br> class EnterpriseKnowledgeGraph:<br> def __init__(self):<br> self.driver = GraphDatabase.driver(“bolt://localhost:7687”)<br> <br> def add_employee_from_api(self, employee_data):<br> with self.driver.session() as session:<br> # Create employee node<br> session.run(“””<br> CREATE (e:Employee {<br> id: $id,<br> name: $name,<br> email: $email,<br> department: $dept<br> })<br> “””, <br> id=employee_data[‘id’],<br> name=employee_data[‘name’], <br> email=employee_data[‘email’],<br> dept=employee_data[‘department’]<br> )<br> <br> def create_reporting_relationship(self, emp_id, manager_id):<br> with self.driver.session() as session:<br> session.run(“””<br> MATCH (e:Employee {id: $emp_id})<br> MATCH (m:Employee {id: $mgr_id})<br> CREATE (e)-[:REPORTS_TO]-&gt;(m)<br> “””, emp_id=emp_id, mgr_id=manager_id)<br> <br> def find_team_structure(self, department):<br> with self.driver.session() as session:<br> result = session.run(“””<br> MATCH (e:Employee {department: $dept})-[:REPORTS_TO]-&gt;(m:Employee)<br> RETURN e.name as employee, m.name as manager<br> “””, dept=department)<br> <br> return [{“employee”: record[“employee”], <br> “manager”: record[“manager”]} for record in result]</p><p><strong>When to Choose Knowledge Graphs</strong></p><p><strong>Choose Knowledge Graphs when you need:</strong></p><ul><li>Explainable AI decisions for compliance</li><li>Complex multi-step reasoning across business relationships</li><li>Integration of multiple enterprise data sources</li><li>Complete, accurate answers to business questions</li><li>Dynamic updates without system rebuilds</li></ul><p><strong>Choose Vector Databases when you need:</strong></p><ul><li>Fast similarity search across large document collection</li><li>Semantic search and content recommendations</li><li>Simple question-answering with good-enough accuracy</li><li>Lower complexity implementation</li></ul><p>Knowledge graphs represent the next evolution in enterprise data management, moving from isolated data silos to interconnected, intelligent systems that mirror how businesses actually operate. While they require more upfront investment than traditional approaches, they provide the foundation for truly intelligent enterprise applications.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a182b9aca8a2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python — FastAPI]]></title>
            <link>https://medium.com/@surezms/python-fastapi-1c2e9c4965f2?source=rss-630dfd05eee0------2</link>
            <guid isPermaLink="false">https://medium.com/p/1c2e9c4965f2</guid>
            <dc:creator><![CDATA[Sureshkumar Mani]]></dc:creator>
            <pubDate>Mon, 18 Aug 2025 18:39:09 GMT</pubDate>
            <atom:updated>2025-08-18T18:39:09.326Z</atom:updated>
            <content:encoded><![CDATA[<h3>Python — FastAPI</h3><h3>🚀 Building Modern APIs with Python FastAPI: A Beginner’s Guide</h3><p>APIs are the backbone of modern applications, powering everything from mobile apps to global enterprise systems.</p><p><strong>What is FastAPI?</strong></p><p>FastAPI is a modern, high-performance web framework for building APIs in Python, created by <strong>Sebastián Ramírez</strong>.</p><p>Built on <strong>Starlette</strong> (for web handling) and <strong>Pydantic</strong> (for data validation), it delivers high performance.</p><p>It’s designed to be:</p><ul><li><strong>Easy to Use</strong> 🤝 — With intuitive syntax and automatic API docs, beginners can get started quickly.</li><li><strong>Type-friendly</strong> 🧩 — It uses modern Python type hints, making your code more readable, reliable, and less error-prone.</li><li><strong>Automatic Documentation</strong> 📖 — Swagger UI and ReDoc are generated for free.</li></ul><p>FastAPI redefining how we build APIs. Its simplicity, speed, and built-in validation make it one of the best choices for modern applications.</p><h3>🛠️ Getting Started</h3><p>First, install FastAPI and the Uvicorn server:</p><pre>pip install fastapi uvicorn</pre><h3>✨ A Simple “Hello World” Example</h3><pre>from fastapi import FastAPI</pre><pre>app = FastAPI()</pre><pre>@app.get(&quot;/&quot;)<br>def read_root():<br>    return {&quot;message&quot;: &quot;Hello, FastAPI!&quot;}</pre><p>Run the app:</p><pre>uvicorn main:app --reload</pre><p>Now visit http://127.0.0.1:8000 and you’ll see your JSON response. Even better, check out http://127.0.0.1:8000/docs — Swagger UI API docs are <strong>ready out of the box</strong>!</p><h3>📦 Handling Path &amp; Query Parameters</h3><p>FastAPI automatically parses variables from the URL or query string:</p><pre>@app.get(&quot;/items/{item_id}&quot;)<br>def read_item(item_id: int, q: str | None = None):<br>    return {&quot;item_id&quot;: item_id, &quot;query&quot;: q}</pre><ul><li>/items/10 → returns { &quot;item_id&quot;: 10, &quot;query&quot;: null }</li><li>/items/10?q=hello → returns { &quot;item_id&quot;: 10, &quot;query&quot;: &quot;hello&quot; }</li></ul><h3>✅ Automatic Data Validation with Pydantic</h3><p>One of FastAPI’s features is Pydantic-based validation. Let’s create a data model for an item:</p><pre>from pydantic import BaseModel</pre><pre>class Item(BaseModel):<br>    name: str<br>    price: float<br>    is_offer: bool = False</pre><pre>@app.post(&quot;/items/&quot;)<br>def create_item(item: Item):<br>    return {&quot;item&quot;: item}</pre><p>If a client sends invalid data, FastAPI automatically returns a structured validation error — no extra code required.</p><h3>🧩 Why Developers Love FastAPI</h3><ol><li><strong>Developer Experience</strong> — Type hints + editor autocomplete make coding faster.</li><li><strong>Performance</strong> — Comparable to Node.js and Go thanks to asynchronous support.</li><li><strong>Scalability</strong> — Perfect for both small prototypes and enterprise production systems.</li><li><strong>Automatic Docs</strong> — No need to write separate API documentation.</li></ol><h3>🔮 When to Use FastAPI?</h3><ul><li>Building <strong>REST APIs</strong> in python for web or mobile clients.</li><li>Designing <strong>microservices</strong> that need fast, async handling.</li><li>Prototyping new ideas quickly with minimal boilerplate.</li><li>Creating <strong>machine learning model wrappers</strong> for deployment.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1c2e9c4965f2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Integration of OpenAI Chat Completions API with Spring Boot]]></title>
            <link>https://medium.com/@surezms/integration-of-openai-chat-completions-api-with-spring-boot-24f7291c0174?source=rss-630dfd05eee0------2</link>
            <guid isPermaLink="false">https://medium.com/p/24f7291c0174</guid>
            <dc:creator><![CDATA[Sureshkumar Mani]]></dc:creator>
            <pubDate>Mon, 28 Jul 2025 18:48:22 GMT</pubDate>
            <atom:updated>2025-07-29T10:13:42.744Z</atom:updated>
            <content:encoded><![CDATA[<p>Integrating the OpenAI Chat Completions API into a Spring Boot application enables us to build smart, conversational assistants and AI-powered features that can handle both free-form dialogue and structured business logic. Here’s how you can approach this integration:</p><h3>1. Direct Integration: Calling OpenAI from Spring Boot</h3><p>The most straightforward method is to make POST requests to OpenAI’s API using Spring’s HTTP client. Here’s how it’s done .</p><h3>Step-by-Step</h3><h3>1. Obtain the API Key</h3><ul><li>Sign up at <a href="https://platform.openai.com/">OpenAI</a> and generate an API key.</li><li>Store it securely (use environment variables or application.properties).text</li></ul><pre># application.properties<br>spring.ai.openai.api-key=YOUR_OPENAI_KEY</pre><h3>2. Set Up the REST Client</h3><p>Use RestTemplate or WebClient to call external APIs:</p><pre>@Service<br>public class ChatService {<br>    @Value(&quot;${spring.ai.openai.api-key}&quot;)<br>    private String apiKey;</pre><pre>    private final ObjectMapper objectMapper = new ObjectMapper();<br>    private final RestTemplate restTemplate = new RestTemplate();</pre><pre>    public String chat(String userMessage) {<br>        try {<br>            // Build request body<br>            ObjectNode requestBody = objectMapper.createObjectNode();<br>            requestBody.put(&quot;model&quot;, &quot;gpt-3.5-turbo&quot;);<br>            requestBody.put(&quot;max_tokens&quot;, 500);<br>            requestBody.put(&quot;temperature&quot;, 0.7);</pre><pre>            ArrayNode messages = requestBody.putArray(&quot;messages&quot;);<br>            // System and user messages...<br>            // (Refer to your code for full construction)</pre><pre>            // Headers and API call<br>            HttpHeaders headers = new HttpHeaders();<br>            headers.setContentType(MediaType.APPLICATION_JSON);<br>            headers.setBearerAuth(apiKey);</pre><pre>            HttpEntity&lt;String&gt; request = new HttpEntity&lt;&gt;(requestBody.toString(), headers);</pre><pre>            ResponseEntity&lt;String&gt; response = restTemplate.postForEntity(<br>                &quot;https://api.openai.com/v1/chat/completions&quot;, <br>                request, String.class);</pre><pre>            JsonNode responseJson = objectMapper.readTree(response.getBody());<br>            return responseJson.get(&quot;choices&quot;).get(0).get(&quot;message&quot;).get(&quot;content&quot;).asText();<br>        } catch (Exception e) {<br>            // Handle API error gracefully<br>            return &quot;I&#39;m having trouble connecting to the AI service right now. Please try again later.&quot;;<br>        }<br>    }<br>}</pre><p>Highlights:</p><ul><li>We have full control over the payload (e.g., model, max_tokens, temperature).</li><li>Error handling is important to ensure user-friendly messages even when the backend is down.</li></ul><h3>2. Spring AI Integration (Function Calling and Orchestration)</h3><p>Spring AI makes it easier to build and expose domain-specific logic as “functions” that our AI model can call, letting us combine language model reasoning with our own business logic.</p><h3>How it Works</h3><ul><li>Define beans/functions for key operations (availability, appointment booking, etc.).</li><li>Spring AI can automatically orchestrate these as “function calls” when the AI model determines they’re needed.</li></ul><h3>Example AIConfig.java (Doctor&#39;s assistant app)</h3><pre>@Configuration<br>public class AIConfig {</pre><pre>    @Bean<br>    public Function&lt;Map&lt;String, String&gt;, Map&lt;String, Object&gt;&gt; checkDoctorAvailability(DoctorService doctorService) {<br>        return input -&gt; {<br>            String doctorName = input.get(&quot;doctorName&quot;);<br>            String dateTimeStr = input.get(&quot;dateTime&quot;);<br>            Doctor doctor = doctorService.getDoctorByName(doctorName);<br>            boolean available = false;<br>            if (doctor != null &amp;&amp; dateTimeStr != null) {<br>                LocalDateTime slot = LocalDateTime.parse(dateTimeStr);<br>                available = doctorService.isDoctorAvailable(doctor, slot);<br>            }<br>            return Map.of(&quot;available&quot;, available);<br>        };<br>    }</pre><pre>    // More functions (e.g., `bookAppointment`) can be defined similarly<br>}</pre><p>Advantages:</p><ul><li>Lets the AI assistant interact directly with backend logic.</li><li>Reduces the need for brittle intent parsing and manual command handlers.</li></ul><h3>3. Hybrid Approach (Recommended in Real Applications)</h3><p>Combine both:</p><ul><li>Use custom logic for routine or simple queries (cheaper, faster).</li><li>Use OpenAI completions for free-form or complex questions.</li></ul><h3>Sample Dispatcher (ChatService.java):</h3><pre>public String chatWithoutCompletionApi(String userMessage) {<br>    String lowerMessage = userMessage.toLowerCase();<br>    if (lowerMessage.contains(&quot;appointment&quot;)) {<br>        return handleAppointmentQuery(userMessage); // Custom logic (no API call)<br>    } else {<br>        return chat(userMessage); // Calls OpenAI API<br>    }<br>}</pre><h3>4. Best Practices</h3><ul><li>API Security: Never expose keys in code repositories; use environment variables.</li><li>Response Filtering: Always check AI-generated content before exposing to users in production.</li><li>Error Handling: Show helpful error messages.</li><li>Observability: Track API usage to manage costs and identify common queries.</li></ul><h3>Conclusion</h3><p>Integrating the OpenAI Chat Completions API with Spring Boot is straightforward. Start simple by invoking the API directly; then, as our app grows, leverage Spring AI for richer workflows and orchestration.</p><h3>Next Steps</h3><ol><li>Set up a Spring Boot project (using Spring Initializr, Maven/Gradle).</li><li>Add dependencies: spring-boot-starter-web, Jackson, and your HTTP client of choice.</li></ol><p>3. Implement direct API call logic for OpenAI.</p><p>(Optionally) Add Spring AI functions to handle structured business actions.</p><p>4.Secure your configuration and test!</p><p>References:</p><ul><li><a href="https://platform.openai.com/docs/guides/gpt">OpenAI Chat Completions API Docs</a></li><li><a href="https://docs.spring.io/spring-ai/reference/">Spring AI Project</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=24f7291c0174" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to LLMs and Generative AI]]></title>
            <link>https://medium.com/@surezms/introduction-to-llms-and-generative-ai-d4d4dff475f0?source=rss-630dfd05eee0------2</link>
            <guid isPermaLink="false">https://medium.com/p/d4d4dff475f0</guid>
            <dc:creator><![CDATA[Sureshkumar Mani]]></dc:creator>
            <pubDate>Mon, 09 Jun 2025 12:46:56 GMT</pubDate>
            <atom:updated>2025-06-10T04:20:54.991Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>What are Large Language Models (LLMs)?</strong></p><p>Large Language Models are advanced artificial intelligence systems trained on vast datasets of textual data. LLM’s designed to understand, generate, and manipulate human language with high proficiency.</p><p>Examples : OpenAI’s GPT series, Google’s Bard, and Meta’s Llama.</p><p><strong>Key Characteristics of LLMs:</strong></p><p><strong>Training Data: </strong>Trained on diverse datasets including books, articles, websites</p><p><strong>Pretraining &amp; Fine-tuning: </strong>Initially trained on general data (pretraining), then specialized for specific tasks (fine-tuning).</p><p><strong>Scale: </strong>Comprise billions to trillions of parameters.</p><p><strong>Training and Inference Process :</strong></p><p>Data Collection: Curate massive datasets.</p><p>Tokenization: Break text into smaller units (tokens).</p><p>Pretraining: Train the model to predict the next token (language modeling).</p><p>Fine-tuning: Adjust the model for specific tasks or domains.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*R2Vsd5aTiud27toLglt-wg.png" /></figure><p><strong>What is Generative AI?</strong></p><p>Generative AI refers to models that can produce new content such as text, images, audio, and video. They learn the underlying patterns of data to generate realistic and coherent outputs.</p><p><strong>Underlying Technologies:</strong></p><p><strong>Transformer Architecture:</strong></p><p>LLMs and Gen AI systems are based on transformers, a neural network architecture introduced by Vaswani et al. in 2017.</p><p><strong>Key features:</strong></p><p>Self-attention Mechanism: Enables models to weigh the importance of different words in context.</p><p>Parallel Processing: Facilitates training on large datasets efficiently.</p><p><strong>Types of Generative AI:</strong></p><p>Text Generation Models: GPT, T5</p><p>Image Generation Models: DALL·E, Midjourney.</p><p>Audio/Video Generation: StyleGAN for images, various deepfake techniques.</p><p>Project lifecyle of Generative AI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/658/1*DAK_3nfg33Zv3ZpeLaxvNw.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d4d4dff475f0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Neural Networks and Generative AI : Revolutionizing Modern Technology]]></title>
            <link>https://medium.com/@surezms/neural-networks-and-generative-ai-revolutionizing-modern-technology-51b0721cb5c2?source=rss-630dfd05eee0------2</link>
            <guid isPermaLink="false">https://medium.com/p/51b0721cb5c2</guid>
            <dc:creator><![CDATA[Sureshkumar Mani]]></dc:creator>
            <pubDate>Sun, 01 Jun 2025 17:30:51 GMT</pubDate>
            <atom:updated>2025-06-02T14:13:38.818Z</atom:updated>
            <content:encoded><![CDATA[<h3>Neural Networks and Generative AI : Revolutionizing Modern Technology</h3><p>In recent years, advancements in artificial intelligence (AI) have transformed various industries, ranging from healthcare to entertainment. At the core of many of these breakthroughs are neural networks and generative AI.</p><p><strong>What Are Neural Networks?</strong></p><p>Neural networks are computational models inspired by the human brain’s interconnected neuron structure. They consist of layers of nodes (or neurons) that process input data, identify patterns, and make decisions or predictions.</p><p><strong>Structure: </strong>Typically arranged into an input layer, one or more hidden layers, and an output layer.<br><strong>Learning Process: </strong>Through a process called training, neural networks adjust connection weights to improve accuracy based on data inputs and known outputs, employing algorithms like backpropagation.</p><p><strong>Types of Neural Networks:</strong></p><p><strong>Feedforward Neural Networks</strong>: Data moves in one direction from input to output.<br><strong>Convolutional Neural Networks (CNNs)</strong>: Designed for processing visual data.<br><strong>Recurrent Neural Networks (RNNs):</strong>Suitable for sequential data like text or speech.</p><p>Neural network depiction with multiple hidden layers:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bfIALH6QJf2yK84fPqcQIw.png" /></figure><p><strong>Generative AI: Creating New Content</strong></p><p>Generative AI refers to systems capable of producing new data that resembles existing data. It leverages neural networks to generate text, images, audio, and video.</p><p><strong>Key Techniques in Generative AI</strong></p><p><strong>Generative Adversarial Networks (GANs):</strong> Consist of two networks — the generator and discriminator — that train together, resulting in highly realistic generated data (e.g., deepfake images, art).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/1*z2kz_LMog0HJQkPCgwKKHw.png" /></figure><p><strong>Variational Autoencoders (VAEs):</strong>Encode data into a compressed form and generate new data by sampling from this representation.</p><p><strong>Transformers: </strong>Architectures like GPT (Generative Pre-trained Transformer) have revolutionized natural language generation.</p><p><strong>Key components of Tranformer architecture:</strong></p><ul><li><strong>Input Text:</strong> The raw text sequence.</li><li><strong>Token Embedding &amp; Positional Encoding:</strong> Converts tokens into vectors and adds positional info.</li><li><strong>Encoder Stack:</strong> Attends to input sequence, capturing relationships.</li><li><strong>Decoder Stack:</strong> Generates output tokens based on encoder outputs and previously generated tokens.</li><li><strong>Output Tokens:</strong> The generated sequence (e.g., translated text, summarized content).</li></ul><p><strong>Applications of Neural Networks and Generative AI</strong></p><p>Natural Language Processing (NLP): Chatbots, translation, summarization.<br>Image and Video Synthesis:Creating realistic images, avatars, deepfakes.<br>Music and Art Generation: Composing music, generating artwork.<br>Healthcare: Drug discovery, medical imaging analysis.</p><p>Neural networks and generative AI are at the forefront of AI innovation, enabling machines to understand, learn, and create with unprecedented sophistication.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=51b0721cb5c2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kafka Transactions and Databases: Achieving End-to-End Data Consistency]]></title>
            <link>https://medium.com/@surezms/kafka-transactions-and-databases-achieving-end-to-end-data-consistency-7446795822b7?source=rss-630dfd05eee0------2</link>
            <guid isPermaLink="false">https://medium.com/p/7446795822b7</guid>
            <dc:creator><![CDATA[Sureshkumar Mani]]></dc:creator>
            <pubDate>Mon, 12 May 2025 18:33:56 GMT</pubDate>
            <atom:updated>2025-05-12T18:33:56.285Z</atom:updated>
            <content:encoded><![CDATA[<p>This article explores how Kafka Transactions can be used to achieve end-to-end consistency when your data pipeline involves both Kafka topics and database operations. We’ll cover the fundamental concepts, common use cases, and provide practical examples.</p><p>In modern data architectures, Apache Kafka often acts as a central nervous system, connecting various microservices and data systems, ensuring data consistency across these systems becomes critical. This is where Kafka Transactions come into play, particularly when integrating with databases.</p><p><strong>The Challenge: Data Inconsistency</strong></p><p>Imagine an e-commerce application where a user places an order. This event might trigger several actions:</p><p>Writing to a Kafka topic: To notify downstream services (e.g., inventory management, shipping)</p><p>Updating a database: To record the order details, customer information, and payment status</p><p>Without proper coordination, failures can lead to inconsistencies. For example:</p><p><strong>Kafka write succeeds, database write fails</strong>:The downstream services process an order that doesn’t exist in the database</p><p><strong>Database write succeeds, Kafka write fails</strong>: The database reflects a completed order, but downstream services are not notified.</p><p>These inconsistencies can result in lost orders, incorrect inventory levels, and a poor user experience.</p><p><strong>Kafka Transactions: The Solution</strong></p><p>Kafka Transactions provide an atomic way to write to multiple Kafka topics and/or update a transactional data store. Atomicity guarantees that all operations within a transaction either succeed together or are rolled back completely, ensuring data consistency.</p><p><strong>How Kafka Transactions Work:</strong></p><p>At a high level, Kafka Transactions involve the following steps:</p><ol><li><strong>Initialization</strong>: A producer is configured with a `transactional.id`. This ID is crucial for Kafka to track and manage the transaction across sessions.</li><li><strong>Start Transaction</strong>: The producer initiates a transaction using `beginTransaction()`.</li><li><strong>Produce Messages and/or Update Database</strong>:The producer sends messages to Kafka topics and performs database operations.</li><li><strong>Commit or Abort</strong>:<br> If all operations succeed, the producer calls `commitTransaction()`.<br> If any operation fails, the producer calls `abortTransaction()`.</li><li><strong>Transaction Coordinator:</strong>Kafka uses a transaction coordinator to manage the transaction’s state. The coordinator ensures that all partitions involved in the transaction either commit or abort together.</li></ol><p>Example Scenario: Order Processing with Kafka and a Relational Database (PostgreSQL)</p><pre>Let&#39;s illustrate with a simplified order processing scenario:<br><br>KafkaProducer&lt;String, Order&gt; producer = createKafkaProducer();<br>JdbcTemplate jdbcTemplate = createJdbcTemplate(); // For database interaction<br><br>producer.initTransactions();<br><br>try {<br>    producer.beginTransaction();<br><br>    // 1. Write order to Kafka topic<br>    producer.send(new ProducerRecord&lt;&gt;(&quot;orders-topic&quot;, orderId, order));<br><br>    // 2. Update order status in the database<br>    jdbcTemplate.update(&quot;UPDATE orders SET status = &#39;PROCESSING&#39; WHERE id = ?&quot;, orderId);<br><br>    producer.commitTransaction();<br>    System.out.println(&quot;Order processed successfully (Transaction committed)&quot;);<br><br>} catch (Exception e) {<br>    System.err.println(&quot;Error processing order. Aborting transaction: &quot; + e.getMessage());<br>    producer.abortTransaction();<br>    // Handle the exception appropriately (e.g., retry, logging, alerting)<br>} finally {<br>    producer.close();<br>}</pre><p><strong>Explanation:</strong></p><p>The code uses a `KafkaProducer` to send messages to Kafka and a `JdbcTemplate` (from Spring) to interact with the database.</p><p>The `initTransactions()` method must be called once per producer instance.</p><p>The `beginTransaction()`, `commitTransaction()`, and `abortTransaction()` methods control the transaction boundaries.</p><p><strong>Important Considerations:</strong></p><p><strong><em>Idempotent Consumers:</em></strong>To achieve true end-to-end exactly-once semantics, your Kafka consumers must also be idempotent. This means that if a consumer receives the same message multiple times (e.g., after a transaction commit and a subsequent consumer failure), it should only process it once. Strategies for achieving idempotence include using unique message IDs or sequence numbers.</p><p><strong><em>Transaction Timeout:</em></strong> Kafka transactions have a timeout. If a transaction remains open for longer than the configured timeout, Kafka will automatically abort it. Configure `transaction.timeout.ms` appropriately.</p><p><strong><em>Performance Impact:</em></strong> Transactions introduce overhead, so carefully evaluate the performance implications for your specific use case. Optimize batch sizes and transaction timeouts.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7446795822b7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Leveraging UML for Software Design]]></title>
            <link>https://medium.com/@surezms/leveraging-uml-for-software-design-8191df7209ad?source=rss-630dfd05eee0------2</link>
            <guid isPermaLink="false">https://medium.com/p/8191df7209ad</guid>
            <dc:creator><![CDATA[Sureshkumar Mani]]></dc:creator>
            <pubDate>Mon, 28 Apr 2025 18:09:38 GMT</pubDate>
            <atom:updated>2025-04-29T05:52:14.069Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Software Design:</strong></p><p>In software design, HLD (High-Level Design) and LLD (Low-Level Design) are two distinct phases that help structure the development process by providing different levels of detail regarding the architecture and components of the system.</p><p><strong>High-Level Design (HLD)</strong><br>HLD is the initial phase of the software design process. It outlines the system’s architecture and design at a macro level, illustrating how different components interact with each other.</p><p><strong>Focus:</strong><br>It emphasizes the overall structure of the system, including the main components, modules, and their relationships. It does not go in details of the implementation but provides a broad outline.</p><p><strong>Tools/Diagrams:</strong><br> — Architecture diagrams.<br> — Use case diagrams.<br> — Data flow diagrams.</p><p><strong>Low-Level Design (LLD)</strong><br>LLD comes after HLD and involves detailing the design of each component identified in the high-level design. It focuses on the specific implementation of the modules.</p><p><strong>Focus:</strong><br> It specifics how each module will be implemented, including the algorithms, data structures, methods, and classes that will be used.</p><p><strong>Tools/Diagrams:</strong><br> — Class diagrams.<br> — Sequence diagrams.<br> — Activity diagrams.</p><p>In these both HLD and LLD process , we use UML to effectively represent software design.</p><p><strong>UML:</strong></p><p>Unified Modeling Language (UML) is a standardized modeling language that has become a cornerstone in the field of software design and engineering. It provides a comprehensive set of diagrammatic tools that enable software architects and developers to visualize, specify, construct, and document the artifacts of a software system.</p><p>UML’s visual nature allows for the representation of complex system architectures in a clear and intuitive manner. One of the primary benefits of UML is its ability to foster better communication among stakeholders.</p><p>Leveraging UML for software design provides significant advantages in terms of communication, complexity management, and adherence to best practices.</p><p>Here is the list of UML diagrams that you can create to enhance software design process.</p><p>1. <strong>Use Case Diagram</strong>: <br> Illustrates the functional requirements of a system from an end-user perspective.<br><strong>Components</strong>: Actors (users or other systems) and use cases (system functionalities).<br><strong>How to Create</strong>: Identify the main actors and their interactions with the system functions.</p><p>2. <strong>Class Diagram</strong>:<br> Represents the structure of a system using classes and their relationships.<br> <strong>Components</strong>: Classes, attributes, methods, and associations (e.g., inheritance, aggregation).<br> <strong>How to Create</strong>: Define the classes necessary for your system and outline the relationships between them.</p><p>3. <strong>Sequence Diagram</strong>:<br> Shows how objects interact in a particular sequence during a specific scenario.<br> <strong>Components</strong>: Objects, messages, and the time sequence of message exchanges.<br> <strong>How to Create</strong>: Identify the objects involved in a use case and the messages exchanged over time.</p><p>4. <strong>Activity Diagram</strong>:<br> Visualizes the flow of control or data through a process.<br> <strong>Components</strong>: Activities, transitions, decision nodes.<br> <strong>How to Create</strong>: Map out the activities and their transitions in a process scenario.</p><p>5. <strong>State Diagram</strong>:<br> Describes the states an object can be in and the transitions between those states.<br> <strong>Components</strong>: States, transitions, events, and actions.<br> <strong>How to Create</strong>: Identify the possible states of an object and the events that cause transitions between them.</p><p>We can use UML modeling tools such as Lucidchart, StarUML, or Visual Paradigm to create these diagrams. By utilizing these visual designs, we can convey complex ideas more clearly and facilitate better understanding among stakeholders in our software development process.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8191df7209ad" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Compensating Transactions -SAGA]]></title>
            <link>https://medium.com/@surezms/compensating-transactions-saga-5d28d49b2f0c?source=rss-630dfd05eee0------2</link>
            <guid isPermaLink="false">https://medium.com/p/5d28d49b2f0c</guid>
            <category><![CDATA[esg]]></category>
            <category><![CDATA[microservices-pattern]]></category>
            <category><![CDATA[saga-pattern]]></category>
            <category><![CDATA[transactions]]></category>
            <dc:creator><![CDATA[Sureshkumar Mani]]></dc:creator>
            <pubDate>Sun, 13 Apr 2025 19:12:54 GMT</pubDate>
            <atom:updated>2025-04-14T16:03:27.504Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Compensating Transaction:</strong></p><ul><li><strong>Purpose:</strong><br> To revert the changes made by a previously completed transaction when a failure occurs.</li><li><strong>How it works:</strong><br> When a step fails, compensating transactions are executed to undo the effects of the successful steps that came before it.</li></ul><p><strong>SAGA :</strong></p><p>The Saga pattern uses compensating transactions as a mechanism to ensure that a distributed transaction either completes successfully or is rolled back to a consistent state in case of failure.</p><ul><li>A saga is broken down into a sequence of local transactions, each performed by a specific service.</li><li>Each local transaction updates the local state of the service and may send a message to trigger the next transaction in the saga.</li><li>If a local transaction fails, the saga executes compensating transactions to undo the changes made by the previous steps.</li></ul><p><strong>Business Use Case:</strong></p><p><em>Implementing an ESG (Environmental, Social, Governance) framework using the SAGA</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gOWiBEumM-k2R-yaPG_BKg.png" /></figure><p><strong>Example Workflow</strong></p><p>1. An organization decides to launch a new sustainability project.</p><p>2. The orchestrator service initiates the process and calls the Environmental Service to record the project.</p><p>3. The Environmental Service completes its transaction and sends an event back to the orchestrator.</p><p>4. The orchestrator then calls the Social Service to engage stakeholders and promote the initiative.</p><p>5. If any transaction fails, the orchestrator invokes compensating actions on previously completed transactions to ensure consistency.</p><p><strong>Implementation Steps</strong></p><ol><li><strong>Identify Services</strong> : Break down the ESG framework into distinct microservices, each responsible for a specific aspect of ESG. For example:</li></ol><ul><li><strong>Environmental Service</strong>: Manages environmental impact assessments, carbon footprint tracking, and sustainability initiatives.</li><li><strong>Social Service</strong>: Handles community engagement, social responsibility programs, and stakeholder relations.</li><li><strong>Governance Service</strong>: Oversees compliance with regulations, internal policies, and governance metrics.</li></ul><p>2. <strong>Design SAGA Orchestration</strong></p><p><strong>Orchestrator Service</strong> : Design a central orchestrator service responsible for coordinating the SAGA transactions across the different microservices. This service will:</p><p>- Initiate ESG workflows (e.g., launching a new sustainability initiative).</p><p>- Manage the sequence of commands sent to each microservice.</p><p>3. <strong>Define Local Transactions</strong></p><p><strong>Local Business Logic:</strong> Each microservice implements local transactions that handle specific operations related to ESG activities. For example, the Environmental Service may have a transaction to log carbon reduction efforts.</p><p><strong>Compensation Mechanisms:</strong> Define compensating actions for each local transaction, ensuring that if a transaction fails, appropriate rollback actions are performed (e.g., if community engagement fails, undo any related interactions logged).</p><p>Sample Code:</p><pre>@Service<br>public class ESGSagaService {<br><br>    @Autowired<br>    private EnvironmentalService environmentalService;<br><br>    @Autowired<br>    private SocialService socialService;<br><br>    @Autowired<br>    private GovernanceService governanceService;<br><br>    public void executeSaga(Order order) {<br>        try {<br>            // Step 1: log project<br>            environmentalService.logProject(order.getEventId(), order.getProjectId());<br>            // Step 2: engage stakeholders<br>            socialService.engageStakeHolders(order.getProjectDetails());<br>            // Step 3: verify compliance<br>            governanceService.verifyCompliance(order);<br>        } catch (EnvironmentalException | SocialException | GovernanceException e) {<br>            // Handle compensating transactions<br>            compensateTransaction(order);<br>            throw e;<br>        }<br>    }<br><br>    private void compensateTransaction(Order order) {<br>        // Compensate project log<br>        environmentalService.unlogProject(order.getEventId(), order.getProjectId());<br>        socialService.dismissStakeHolders(order.getProjectDetails());<br>    }<br>}<br><br>// Placeholder methods for services<br>class EnvironmentalService {<br>    void logProject(String eventId, String projectId) { /* ... */ }<br>    void unlogProject(String eventId, String projectId) { /* ... */ }<br>}<br><br>class SocialService {<br>    void engageStakeHolders(ProjectDetails projectDetails) { /* ... */ }<br>    void dismissStakeHolders(ProjectDetails projectDetails) { /* ... */ }<br>}<br><br>class GovernanceService {<br>    void verifyCompliance(Order order) { /* ... */ }<br>}<br><br>// Placeholder domain classes<br>class Order {<br>    private String eventId;<br>    private String projectId;<br>    private ProjectDetails projectDetails;<br>    // Getters and setters<br>}<br><br>class ProjectDetails {<br>    // Getters and setters<br>}</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5d28d49b2f0c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>