<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Praveen Manvi on Medium]]></title>
        <description><![CDATA[Stories by Praveen Manvi on Medium]]></description>
        <link>https://medium.com/@pmanvi?source=rss-9398dbbcf83a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*NUDvOIpa5TshnYjMZcXaFQ.jpeg</url>
            <title>Stories by Praveen Manvi on Medium</title>
            <link>https://medium.com/@pmanvi?source=rss-9398dbbcf83a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 19:46:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@pmanvi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Style Guide for Governed Autonomy: AI-Native Architecture for Enterprise Systems]]></title>
            <link>https://pmanvi.medium.com/style-guide-for-ai-native-architecture-59706dbc3467?source=rss-9398dbbcf83a------2</link>
            <guid isPermaLink="false">https://medium.com/p/59706dbc3467</guid>
            <dc:creator><![CDATA[Praveen Manvi]]></dc:creator>
            <pubDate>Wed, 25 Mar 2026 10:57:12 GMT</pubDate>
            <atom:updated>2026-10-11T12:59:14.169Z</atom:updated>
            <content:encoded><![CDATA[<h4>A practitioner’s methodology for building governed, evolvable AI systems</h4><p>This document tries to come up a paradigm to adopt agentic system with the foundtions in IAM and Orchestration engine in an enterprise setup. We are replacing <strong>deterministic software flows</strong> with <strong>probabilistic, autonomous actors</strong>. This is not well understood although its clear that AI juggernaut is impossible to stop. Unbounded agents with unclear authority and no governance model leads to Hidden side effects, Security risks, Compliance nightmares, Debugging impossibility and Lack of trust. Customer trust hinges on enabling <strong>explainable, guardrailed automation</strong> and <strong>human-in-the-loop controls</strong> that build user confidence. Over time, these features will facilitate the transition to trusted AI actions without human oversight.”</p><blockquote>In agentic systems, orchestration is not just about sequencing tasks — it is about governing who (agent identity), what (entitlements), and how (permissions) before any action is executed. This chain becomes the backbone of safe autonomy, with human supervision as the final control plane. We are in phase where Software is no longer just executed. It is <em>authorized, orchestrated, and supervised.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7KgnNkd4lGEh-GtkpVI-9w.png" /></figure><p><strong>AI Architecture Needs a Spine, Not Just a Model</strong></p><p>Most teams/projects don’t fail with AI because the model is weak. They fail because the surrounding system is vague. When AI is dropped into a product without structure, everything starts to wobble: outputs vary, costs drift, approvals become fuzzy, and nobody can explain why one decision was made over another. The fix is not another prompt tweak. The fix is architecture.</p><p>The central idea in the this document is to treat AI as part of a governed system, not as a magical layer on top of one.</p><ul><li>Cognitive loop: model plans, calls tools, observes, retries, delegates to subagents.</li><li>Control loop: workflow engine manages state, permissions, approvals, validations, audit, and final writes.</li></ul><p><strong>Start with orchestration, not the model</strong></p><p>The most important design choice is simple: the model should not be the center of the application. It can be the center of reasoning inside a bounded task loop but orchestration should be. Camunda like solution play very important role here. AI systems need context architecture, not just vector storage: memory tiers, retrieval policy, compaction, source citation, session state, and audit retention</p><p>AI is probabilistic by nature. Business operations are not. If pricing, targeting, planning, or approvals depend on AI, then the workflow around those decisions must still be deterministic, observable, and auditable. Enterprise cannot run heuristic mode.</p><p>That means every meaningful AI action should sit inside a workflow with clear state transitions, validation steps, fallback paths, and human review where needed. AI can generate recommendations, extract signals, and propose next actions. It should not silently mutate the system of record.</p><p>In practical terms, this creates a cleaner contract: AI provides cognition, while the workflow engine provides control.</p><p><strong>Build in layers so responsibilities stay clear</strong></p><p>AI architecture separates concerns into five layers:</p><blockquote>1. AI gateway services for model routing, resilience, and guardrails.</blockquote><blockquote>2. Orchestration for agents, workflows, tools, and business process control.</blockquote><blockquote>3. UI integration for APIs, streaming responses, and frontend contracts.</blockquote><blockquote>4. UX standards for loading states, confidence display, and user feedback.</blockquote><blockquote>5. Data and vector strategy for search, analytics, caching, and audibility.</blockquote><p>This matters because AI systems become fragile when one layer tries to do everything. The backend should not own UX trust signals. The model provider should not define business rules. The frontend should not call models directly. Strong AI products come from clean boundaries.</p><p>The use of UX exists in the initial phase and as agents becomes more autonumous and the UI purpose limited only to handle the escalation paths and monitor where human intervention is required.</p><p><strong>Treat AI like any other critical dependency</strong></p><p>A mature AI system should be engineered with the same discipline used for databases, payment providers, or core infrastructure.</p><p>That means every model call needs retries, timeouts, circuit breakers, rate limits, and fallback paths. It means token usage and estimated cost should be tracked as first-class telemetry. It means teams should choose models intentionally by task instead of defaulting to the most powerful option everywhere.</p><p>One useful mindset shift is this: temperature is not just a tuning trick. It is part of the contract. Extraction and classification should run at low variance. Analysis can tolerate some flexibility. Generative work can allow more range. Once teams define that clearly, testing and operations become much more stable.</p><p><strong>Agents should compete for work, not for authority</strong></p><p>Agentic systems become dangerous when they are allowed to act without clear boundaries. A better pattern is brokered execution.</p><p>Instead of letting multiple agents write directly into business objects, publish a task through a workflow. Let eligible agents evaluate it. Select one execution path based on capability, cost, and confidence. Then validate the structured output before anything is committed.</p><p>This keeps the system safe while still unlocking speed. Agents contribute reasoning. The platform keeps authority.</p><p>That separation is especially important in environments where approvals, compliance, or customer-facing decisions are involved. High-confidence outputs may move quickly. Lower-confidence outputs should route to a human. Confidence is not decoration; it is a decision input.</p><p><strong>User trust is part of the architecture</strong></p><p>Many AI systems focus heavily on prompts and almost ignore experience design. That is a mistake.</p><p>If an AI action takes ten seconds, the interface should explain what is happening. If an answer is uncertain, the user should see that uncertainty. If content is AI-generated, it should be labeled. If retrieval is involved, sources should be visible. If the model gets something wrong, there should be a fast feedback loop.</p><p>Trust does not come from saying the system is intelligent. It comes from making the system legible.</p><p>That is why confidence badges, review prompts, citations, editable outputs, and meaningful error states are not just UX polish. They are architectural requirements for responsible AI products.</p><p><strong>Be pragmatic about data, not ideological</strong></p><p>There is no reason to force every AI workload into the same storage model.</p><p>Use PostgreSQL and pgvector when they are sufficient. Use Redis for session speed. Use analytical systems for large-scale reporting. Use graph stores when the problem is actually graph-shaped. Start with the simplest stack that meets current needs, then move to specialized infrastructure only when there is real evidence that scale or latency requires it.</p><p>The same principle applies to retrieval pipelines. Chunking, metadata, citations, and audit trails matter more than trendy tooling. A grounded system with modest infrastructure is far more valuable than an overengineered one that no team can operate.</p><p><strong>The real goal is governed autonomy</strong></p><p>The promise of AI in enterprise systems is not just faster output. It is staged autonomy. Teams usually begin with AI as an assistant: draft this, classify that, recommend the next move. Over time, if the platform captures confidence, audit history, performance, and approval patterns, some tasks can shift from assisted to semi-autonomous. Eventually, a subset can become fully automated with humans focused on exceptions and strategy. That progression only works when the architecture is designed for it from the beginning.</p><p>AI products do not become reliable because the model gets smarter. They become reliable because the system around the model gets stricter. That is the core lesson of the stylesheet: if you want AI in production, design the control plane first.</p><h4>Why Embabel as agent framework</h4><p>The team behind Spring is tackling this head-on with Embabel, an open-source framework that brings a “Java-first” approach to AI agents and it ensures that Production-Ready Foundation is available. Our platform is built on Spring, and our teams are fluent in its patterns. Forcing a Python-based framework into this stack creates an unnecessary operational boundary. Embabel introduces true planning algorithms rather than relying on finite state machines or sequential execution.<br>Seamless Fit &amp; Unified Ecosystem: Embabel is built <em>with</em> Spring and Kotlin.<br>Compile-Time Guarantees and Preventing Runtime Surprises has been corner stone of building determenestic solutions in the enterprise making JVM is choice of running environment. For machine critical systems Embabel is a compelling alternative to python based tools like LangGraph, Pydantic AI, and CrewAI for building deterministic agents on the Spring platform. Unlike Python frameworks where data contracts rely on runtime validation (Pydantic) or TypedDicts, Embabel provides compile-time checks, refactoring support, and IDE assistance that natively integrates with JVM eco system.</p><p>Although Spring founder’s involvement guarantees architectural maturity; v0.3.3+ demonstrates active, structured development, its still newer compared to pythonic alternatives and relatively smaller eco system.</p><h3><strong>Agentic User Onboarding: From Manual Review to Governed Autonomy</strong></h3><blockquote>Here is one practical example of AI-native architecture ‘user onboarding’ that we have implemented.</blockquote><p>In many enterprise systems, onboarding looks simple from the outside: a user registers, someone reviews the request, assigns roles, and maybe schedules an introductory meeting. But operationally, this is rarely trivial. The reviewer needs to understand the request, classify the user, choose the correct entitlement, avoid over-provisioning, follow policy, and leave enough audit history for security and compliance.</p><p>This is exactly the kind of workflow where an agentic system helps, as long as the agent is not treated as an uncontrolled automation script.</p><p>The better pattern is to keep the onboarding process inside a workflow engine and introduce a specialized agent as a participant in that workflow.</p><p>In this model, the workflow still owns the process. User tasks, approvals, rejections, service calls, retries, and escalations remain deterministic. The agent does not replace the process. It listens to onboarding tasks assigned to a specialist group, evaluates the request, and proposes the next best action.</p><p>For example, an onboarding specialist agent can:</p><ol><li>Inspect the registration request.</li><li>Recommend the correct role or entitlement.</li><li>Determine whether the role is low, medium, or high sensitivity.</li><li>Suggest calendar setup details for follow-up.</li><li>Write rationale, confidence, and evidence back into the workflow.</li><li>Allow a human reviewer to approve, adjust, or reject the recommendation.</li></ol><p>The important part is that the agent’s output becomes structured workflow data, not an invisible side effect.</p><p><strong>Assist, Compete, Autonomous</strong></p><p>This pattern supports staged autonomy.</p><p>In the assist phase, the agent only recommends. A human specialist still reviews the task and clicks approve or reject. This immediately reduces manual analysis effort without changing the control boundary.</p><p>In the compete phase, multiple agents or policies can evaluate the same task. One agent may specialize in role classification, another in risk policy, another in calendar coordination. The workflow selects the best outcome based on confidence, policy, and business rules.</p><p>In the autonomous phase, only safe actions are automated. For onboarding, that usually means low-sensitivity approvals with high confidence. High-sensitivity roles still require human review. Automated rejection should be treated even more carefully and often excluded from the first version.</p><p>That progression matters. Autonomy should be earned through evidence, not enabled by enthusiasm.</p><p><strong>Technologies Used</strong></p><p>A practical implementation was combination of:</p><ul><li><strong>Camunda</strong> for BPMN orchestration, human tasks, candidate groups, process variables, retries, and auditability.</li><li><strong>DMN</strong> for task allocation and policy-driven routing.</li><li><strong>Spring Boot</strong> for the agent service, workflow clients, REST integrations, and background polling.</li><li><strong>Spring AI</strong> for model abstraction, prompt execution, and structured AI interaction.</li><li><a href="https://github.com/embabel/embabel-agent"><strong>Embabel</strong></a> for agent modeling, goals, actions, and tool-oriented orchestration.</li><li><a href="https://github.com/embabel/dice"><strong>DICE</strong></a><strong>-style memory/context</strong> for decision context, previous outcomes, and confidence grounding.</li><li><a href="https://github.com/dokimos-dev/dokimos">Dokimos</a> is an evaluation framework to evaluate responses, track quality over time, and catch regressions</li><li><strong>Database history</strong> for agent runs, decisions, confidence scores, fallback reasons, and KPI reporting.</li><li><strong>React/Next.js UI</strong> for showing recommendations, agent action badges, and KPI/log views.</li></ul><p>The UI is not a minor detail here. If an agent touched a task, the user should see it. If the agent recommended a role, the reviewer should see the role, confidence, rationale, and evidence. If the agent completed work automatically, the system should show that explicitly.</p><p>That visibility turns automation into trust.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aD_IbEBn88IdWrw7El-XZw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jvTUDi11CSmHAsfNTNtWsQ.png" /></figure><blockquote>The key idea is simple: every business process becomes a context-producing engine. Each task, approval, override, pricing decision, channel choice, and final outcome becomes reusable institutional memory. platform packages the right slice of that memory into a context capsule for the next agent action.</blockquote><p>That is Context Engineering: not “send a bigger prompt to a model,” but systematically assembling the right business context, prior decisions, human judgment, policy, confidence thresholds, and tool outputs at the exact moment work is being done.</p><p>In the above onboarding use case, this is already visible. The Onboard Specialist Agent recalls prior decisions by applicant domain and agency/brand, separates semantic memory from episodic examples, folds in human reviewer judgments, recommends least-privilege roles, scores confidence, and only advances from Assist to Compete to Autonomous when evidence supports it. High-sensitivity access still routes to human review. The agent gets smarter, but the process stays governed.</p><p><strong>Measuring the Value</strong></p><p>The strongest KPI for this kind of agent is not “number of AI calls.” It is reduction in manual operations.</p><p>Useful metrics include:</p><ul><li>Agent-reviewed tasks.</li><li>Human-reviewed recommendations.</li><li>Auto-completed low-risk approvals.</li><li>Manual operations avoided.</li><li>Estimated reviewer minutes saved.</li><li>Human override rate.</li><li>Confidence distribution by task type.</li><li>Escalations caused by policy or uncertainty.</li></ul><p>This creates a feedback loop. If the agent is often correct for low-risk onboarding, more of that lane can move toward autonomy. If humans frequently override recommendations for a certain role type, that path should stay assisted until the policy or context improves.</p><p>The larger lesson is simple: agentic onboarding works when the agent becomes a governed worker inside the process, not a shortcut around it.</p><p>Workflow provides control. The agent provides cognition. The database provides memory. The UI provides trust.</p><p>That combination is what makes the system evolvable. It can start as assistant-first, grow into supervised automation, and eventually handle a subset of onboarding tasks autonomously without losing auditability or business control.</p><h4>Summary</h4><p>The crux of the idea is that providing independent, focus trained agents is not enough, we need robust, auditable decision-making and orchestration around how and why they’re called upon in enterprise environment where determinism is more important. An orchestrator engine (say Camunda) is essential for achieving this because it already specializes in integrating systems and lets designers tightly control how and why those systems are accessed. Biggest trustworthy actions comes from far better auditability. Combining the data generated from stepping through the various paths of the process with the chain of thought output from each agent gives a complete picture of how and why certain actions were taken.</p><blockquote><em>The path to enterprise AI is not giving agents more freedom by default. It is designing a system where autonomy is earned through evidence. Models can reason, propose, and execute bounded tasks, but workflows decide when an action becomes authoritative. The future architecture is a thin intelligent agent loop inside a strong control plane: observable, auditable, permissioned, and capable of increasing autonomy as trust is proven.</em></blockquote><p><strong>Control Plane (Camunda + IAM + Policy)</strong></p><ul><li>Process orchestration</li><li>Identity &amp; entitlements</li><li>Decision governance</li></ul><p><strong>Execution Plane (Agents + Workers) — </strong><em>These all can ride broad shoulders of well tested Spring eco system</em></p><ul><li>LLM-powered reasoning</li><li>API integrations</li><li>Task execution</li></ul><p><strong>Data Plane</strong></p><ul><li>Context</li><li>Memory</li><li>Observability</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cuPwUmhI8Az90ERDvC9CjQ.png" /><figcaption>Safe Agentic Path for organizations to follow</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2zi6m2L-RZ32LkasY6HfRQ.png" /></figure><p>Camunda 8.9 introduced major capabilities to blend deterministic business processes with dynamic, AI-driven operations:</p><p><strong>Agentic Orchestration:</strong> Seamlessly integrates with AI models and agent frameworks to bridge the gap between “black box” AI and structured corporate governance.</p><ul><li><strong>MCP (Model Context Protocol) Support:</strong> The platform acts as an MCP server, allowing external AI clients (like Claude Code and Claude.ai) to natively investigate, trace, and even fix in-flight incidents using natural language.</li><li><strong>Advanced Connectors:</strong> Includes built-in Agent-to-Agent (A2A) connectors, model configuration upgrades, and process instance migration for in-flight agentic workflows.</li></ul><blockquote><strong>Orchestration is foundational for building agentic software with human in the loop with people oriented business, not optional</strong></blockquote><p>Orchestration isn’t another application feature, it’s the process spine of platform. It centralizes state, coordination, assignment, approvals, exceptions, audit, and visibility. Rebuilding that ourselves would consume significant effort and still risk fragmented, unreliable workflow logic. The strategic choice is to build differentiated business experiences on top of a proven orchestration backbone, not to poorly recreate one. Frankly, I can’t imagine building quote-to-cash without a strong orchestration base underneath it.</p><p>Here’s what we get with it, versus what we’d own forever without it:</p><ol><li><strong>Human work in the process</strong><br> With it: Assignments, approvals, rework, escalations, and SLAs are native to the process model.<br> Without it: We build and maintain approval UIs, queues, notifications, and callback logic indefinitely.</li><li><strong>One shared model for business and IT</strong><br> With it: Both work from the same standards-based BPMN/DMN model in Web Modeler — readable, validatable, and governable by both sides.<br> Without it: Logic lives in code; business can’t read or govern it directly.</li><li><strong>Visible, reusable logic</strong><br> With it: Process and decision logic sits in versioned, standards-based BPMN/DMN models.<br> Without it: Decision logic is buried in code.</li><li><strong>Leadership visibility</strong><br> With it: Optimize surfaces campaign status, handoffs, approval delays, SLA risk, exceptions, and time-to-launch.<br> Without it: Logs show technical execution only; SLAs must be engineered separately.</li><li><strong>Agentic AI with guardrails</strong><br> With it: Agents run inside defined boundaries, business rules, and human checkpoints — with escalation and traceability built into the workflow.<br> Without it: Governance is custom-built and maintained in Step Functions.</li><li><strong>Continuous improvement</strong><br> With it: ProcessOS (beta; GA expected Oct 2026) re-engineers processes against goals like lower SLA, lower cost, and better CX for the agentic era.<br> Without it: Optimization becomes a separate future build.</li></ol><p>Scale:<br>Intuit claims that It can handle 15M processes in 10 hours, ~375 process instances/sec closely followed by Attlassian/JIRA running on camunda.</p><p>Secure:<br>UI/Actions by definition, by design are secure in this eco system with User Task List driving everything. All the UI elements(forms, actions) are generated dynamically based on user assignment, there is no possibility of cross pollination OR developer dependency for permission checking.</p><p>Cost of Change<br>Introducing new changes in data elements, actions, approval flows, assignment rules all can happen without any engineering deployment of binaries. Studio led development brings in agility.</p><p>Agentic Integration<br>In this case we don’t have chat, and is a poor and misleading control panel for autonomous, multi-step AI agents.<br><a href="https://reprint.forrester.com/reports/the-forrester-wavetm-adaptive-process-orchestration-software-q3-2026-4cbbe882/index.html">Forrester report</a> published last why this is credible way to get into agentic and deal with rogue agents.</p><p>To be fair there is no magic here, it took ~15 years of industry standardization on BPMN/DMN and stabilizing the runtime from awesome engineers at camunda.</p><p>Of course there is a real cost — learning curve, licensing, infrastructure, and setting aside some features that don’t fit our product view. That trade is worth it: it buys us a common, visual, governed language for modeling, auditing, and evolving processes, and it lets AI agents operate within clear boundaries without losing control, traceability, or accountability.</p><p><a href="https://camunda.com/solutions/industry/media-and-entertainment/">Process Automation for Media &amp; Entertainment — closest fit for content, ad sales, campaign launches, and AI workflows:</a></p><p><a href="https://camunda.com/case-study/zdf-studios/">ZDF Studios case study — content-rich, multi-system media operations</a>:</p><p><a href="https://camunda.com/case-studies/gema/">GEMA case study — rights management, manual work, and business rules:</a></p><p>References</p><p><a href="https://github.com/kaspel-dev/essayist">https://github.com/kaspel-dev/essayist</a> my teaching application for buildung AI Agents<br>https://codex.danielvaughan.com/2026/04/09/rod-johnson-embabel-jvm-agent-framework/ — Historical perspective and looks into brilliant mind of Rod Johnson <br><a href="https://medium.com/@igor.dayen/embabel-agentic-ai-framework-release-1-0-0-new-features-332f375d0fba">https://medium.com/@igor.dayen/embabel-agentic-ai-framework-release-1-0-0-new-features-332f375d0fba</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=59706dbc3467" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI Agent-Native Development]]></title>
            <link>https://pmanvi.medium.com/beyond-copilots-building-for-the-autonomous-future-a-practical-protocol-for-agent-native-ea067a26c205?source=rss-9398dbbcf83a------2</link>
            <guid isPermaLink="false">https://medium.com/p/ea067a26c205</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[ai-tools]]></category>
            <dc:creator><![CDATA[Praveen Manvi]]></dc:creator>
            <pubDate>Wed, 01 Oct 2025 19:59:46 GMT</pubDate>
            <atom:updated>2026-02-05T06:19:45.451Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jNRLY5kEfm8fn6ve" /></figure><h3>Introduction</h3><p>For the past decade, SaaS has meant building for humans — polished UIs, frictionless workflows, and engagement metrics. But the most important users of software in the next decade won’t be people. They’ll be <strong>AI Agents</strong>. This shift isn’t about sprinkling a chatbot on top of your app. It’s about re-architecting software for a world where agents — not humans — read, decide, and act across systems.</p><p>I think Models will continue to leapfrog each other. Open source models will narrow the gap with big hosted models. And orchestration technologies like spring-ai, pedantic, koog, embabel … and architectural patterns will be key to unlocking business value.</p><p>In this two-part section, I’ll try to unpack the coming paradigm and share a concrete protocol I have used to develop and build agent-native software.</p><h3>1. The Future of SaaS Is Agentic</h3><p>GenAI is evolving from assistive copilots to autonomous, goal-driven systems. Human clicks are being replaced by agent actions.</p><p>Agents won’t log in, navigate menus, or attend training. They’ll consume APIs, evaluate data, and complete workflows end-to-end: “Workflow” is the keyword.</p><p>An underwriting agent evaluates risk by calling compliance APIs.<br>A sales agent qualifies leads and updates CRM pipelines.<br>A compliance agent proactively analyzes logs.<br>A collections agent resolves invoices directly in your financial system.</p><p>The implications are clear — the core user of the system is changing.</p><p>In the New Competitive Landscape, the playing fields are getting levelled for both Legacy SaaS, which are Rich in users, data &amp; workflows. (Salesforce, ServiceNow) API-first, agile, Agent-Native Startups purpose-built (Adept, Fixie) products.</p><p>Winners won’t be those with the most features. They’ll be those who empower the most autonomous, highest-quality agents.</p><p>What should we do? AI juggernaut is unstoppable</p><p>Be Agentic: Expose APIs that are actionable, contextual, and well-documented.<br>Be Agent-Native: Build software as a network of agents. Expose actions, not just data.<br>Monetize Outcomes, Not Usage: Seat-based pricing is irrelevant when agents are the users. Value-based models — % recovered chargebacks, hours saved, revenue gained — become the new standard.</p><h4>Vanity AI vs. Value AI</h4><p>Not all AI is equal.<br>Vanity AI: Chat features bolted onto UIs — novel, but not transformative, that act as productivity tools with no tangible impact.<br>Value AI: AI embedded in critical workflows, driving measurable business outcomes. Most companies are still stuck at vanity. If the solution is a productivity booster living only in a browser, it&#39;s likely to be vanity AI. Those who design for value will define the next era.</p><p>Does it drive a business-critical outcome? (reduces costs, increases revenue …)<br>Can you tie it directly to a new pricing model? (% of value delivered)<br>Are the Product Team and Revenue teams aligned on their measurable impact?</p><p>The above questions should settle the direction and scope.</p><p>AI/Generative AI are vanity projects for CIOs as the individual productivity benefits are subjective hard to prove where as Agentic AI is where workflow, actions exists and where the real revenue comes from.</p><h3>2. A Practical Protocol for Agent-Native Development</h3><p>At my organization, we’ve moved beyond theory. We use AI agents not just in our products, but in the development process itself. Our philosophy: use agents to build the systems that agents will use. We need to move from Copilot to Colleague</p><p>Here’s the two-stage protocol I have refined:</p><p>Stage 1: Generate the Backend &amp; API Contract<br>Goal: A robust, API-first backend as the single source of truth.<br>Tooling: After experimenting with Replit, Bolt, Lovable, Junie, and Copilot, we standardized on Amazon Q for its deep integration with our cloud stack.<br>I guess any tool that’s integrated with an IDE and agentic is more valuable</p><p>My IntelliJ IDEA looks like this, and it makes it a lot simpler than Visual Code-driven tools for Kotlin/Java development. We can mix and match multiple solutions with the same prompt with these tools. It&#39;s quite smooth and seamless as a continuous development process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/134/1*mxbrI2nkRJi4bTxR_B6cxw.png" /><figcaption>Agents</figcaption></figure><h4>Here’s the two-stage protocol I have refined:<br>Stage 1: Backend Born API-First — Generate the Backend &amp; API Contract</h4><p>Goal: Create a robust backend that exists for agents, not UIs. The explain the APIs from persona-based usage. The UX is embedded with an API<br>Tooling: After experimenting with Replit, Bolt, Lovable, Junie, and Copilot, I liked Amazon Q for its deep integration with our cloud stack and being tightly integrated with the IDE.<br>Method: Prompt-driven design → backend generation in our backend framework (Spring Boot) that we have standardized → complete Swagger/OpenAPI spec as the primary output. This is ur blueprint for both humans and agents and serves as the MCP server.<br>Why it works: The backend is “born API-first,” designed for agents from the ground up. The API contract is the single source of truth. Every agent, human or machine, consumes the same blueprint.</p><h4>Stage 2: Frontend as a Reflection, Not a Canvas — Agent-Native Frontend</h4><p>Goal: A UI that reflects the API, usable by humans and agents.<br>Tooling: optimized for translating API contracts into UIs. A UI that mirrors the API — thin, agent-friendly, zero drift.<br>Method: Feed the Swagger spec into an AI agent.<br>Add persona + UX requirements (Ex, ops manager dashboard).<br>The agent builds a Next.js frontend with forms, tables, and actions tied directly to API endpoints. Layer on persona-driven UX prompts. Output a Next.js app (which is our front-end standard) aligned perfectly with the backend.</p><p>Results<br>Speed: Prototype in hours, not weeks.<br>Consistency: No frontend/backend mismatch — the API spec rules everything.<br>Agent-Native: APIs are clean, documented, and action-oriented; UIs are thin layers optimized for both humans and their agents.</p><p>The example here is building an Incident Manager for the workflow engine (Camunda) that we use and take actions, mostly retrying OR cancelling that tasks/activities that are performed in a controlled manner that are auditable and alert-enabled. I will be moving this solution into agentic mode in the coming days with <a href="https://github.com/embabel">Embabel</a> framework. The result was pretty decent.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b62d83f8dd40c39b62c9a08f421fa6d0/href">https://medium.com/media/b62d83f8dd40c39b62c9a08f421fa6d0/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9upzEsJgOCa1xVvwZds70g.png" /></figure><h4>Kiro</h4><p>Kiro comes with robust SDLC for building application,</p><p>Requirement: I have one devops job to push my application running through the docker-compose to production, I know that Kiro replaces traditional coding with natural language specification. We can express what we need in plain English, and Kiro builds working applications that those users can iterate and refine themselves</p><p>Define “requirements.md” -&gt; Creates “design.md” based on requirements and plan for execution and then creates “tasks.md” listing out all the tasks. Key is define requirements in details, design comes with how its getting and tasks gets it done.</p><p>For me this looks more traditional and fits better into large teams compared to claude.md having everything.</p><p>Here is an example that used for depplyment. Kiro shines when everything is in AWS.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2543b9a73c80f6aa7cee22a7121d7b8e/href">https://medium.com/media/2543b9a73c80f6aa7cee22a7121d7b8e/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1qAdvT3bcwy_6aEZV8rRzw.png" /><figcaption>Help me with Tools, Skills, MCPs I will get your job done.</figcaption></figure><h4>How about JVM world?</h4><p>Calling APIs from OpenAI or HuggingFace. We don’t care about tensors, linear algebra, or “how the sausage is made”.</p><p>Spring AI again comes to rescue with ultimate form of “Spring Magic.” Here we can follow the pattern to inject: <a href="http://twitter.com/Autowired">@Autowired</a> ChatClient. Relying on the “convention over configuration” philosophy, it integrates seamlessly with your existing Spring Boot apps. Perfect for whipping up REST endpoints, chatbots, and simple RAG chains.</p><p>We need complex Agents and orchestration we have <a href="https://github.com/embabel">Embabel</a> built on above foundations. 99% of applications need to deal with upper layer only</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/674/1*QLtq_R_XT7a1F12nD7rCpg.png" /></figure><h4>Summary</h4><p>Agent-native development is not a thought experiment — it’s our daily practice. And the results are revolutionary: faster delivery, higher consistency, and systems that are ready for the autonomous future. The next decade of software won’t be built for humans. It’ll be built for AI agents.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ea067a26c205" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Case for using Nomad in Spring Cloud eco system.]]></title>
            <link>https://pmanvi.medium.com/case-for-using-nomad-in-spring-cloud-eco-system-fc315aafe29b?source=rss-9398dbbcf83a------2</link>
            <guid isPermaLink="false">https://medium.com/p/fc315aafe29b</guid>
            <dc:creator><![CDATA[Praveen Manvi]]></dc:creator>
            <pubDate>Thu, 10 Oct 2024 18:27:06 GMT</pubDate>
            <atom:updated>2026-04-04T16:36:47.585Z</atom:updated>
            <content:encoded><![CDATA[<h3>Nomad and Spring Cloud</h3><blockquote>Bringing back Simplicity for running java native workloads</blockquote><p>When orchestrating workloads, we default to containerisation platform (k8s), here this writeup is for more simpler option to consider for simpler Java workloads, especially those that are lightweight, stateless and frugal in their resource usage, an alternative approach like <a href="https://www.nomadproject.io/"><strong>HashiCorp Nomad</strong> </a>can be far more efficient without the added overhead of containers.</p><h4>Minimal Overhead for Non-Docker Workloads</h4><p>Services, built on Spring Cloud, that have characteristics of using minimal CPU and mostly I/O bound, introducing Docker adds an extra layer of abstraction, which results in additional CPU and memory usage. While the overhead isn’t excessive, it is noticeable. For systems where efficiency is<strong> </strong>paramount, avoiding Docker’s additional consumption might be beneficial. Nomad provides the flexibility to run<strong> </strong>Java native JARs directly, thereby skipping the need for containers. This is perfect for Spring Boot applications, which are typically lightweight and self-contained.</p><p>Nomad’s ability to orchestrate without requiring Docker keeps the resource footprint low. You only need to manage and monitor your application workloads, without the extra complexity of managing container layers, image building, or Docker daemons.</p><h4>Frugality in Resource Management</h4><p>Nomad is optimised for environments where <strong>efficiency</strong> and <strong>simplicity</strong> are key. It doesn’t impose unnecessary bloat. In environments where resources need to be allocated sparingly, and the primary concern is running multiple Java services with minimal resource usage, Nomad is the perfect fit and combined with , create a cost-efficient system with less pressure/requirement with SREs.Running Docker containers will consume more resources than running spring-boot JARs directly.<br>Some ballpark numbers about the overhead per service.Memory: ~500MB <br>,CPU: ~15% , Disk Space: ~1GB , Network: ~Few milliseconds (In bridge mode with virtualisation). All the legacy apps (with .war, .ear deployment can be made made to run natively)</p><p>As the frugality of services is essential. These Spring Cloud services (for discovery, observability, and many others that have corresponding implementation in k8s) can run directly on Nomad, preserving their lightweight nature and minimising the need for additional packaging.</p><h4>Scheduling</h4><p>Nomad offers a lightweight scheduling framework. Simplicity comes with siingle binary and offers multi workload support. The frugal resource utilisation with flexible deployment options, Unlike k8s Nomad focusses on job scheduling and doesn’t come up with many dependencies (service mesh, complex networking setup). Scheduling is critical in workload management because it optimises how tasks and applications are executed across available computing resources. Scheduling is essential to optimise resource use, improve performance, handle failures, scale applications, and ensure cost efficiency in modern computing environments.</p><h4>Operational Simplicity</h4><p>Nomad’s single binary and simplicity in operations make it a perfect choice for teams looking to reduce their operational burden. The complexity of some orchestrators can be overkill for environments that don’t need advanced scaling or networking. Nomad provides a simple interface for scheduling, deploying, and managing Java applications, which is ideal for teams prioritising operational frugality and simplicity.</p><h4>Show me code please?</h4><p>Here is example to get the feel of what it takes to build services and with Spring cloud eco system and Nomad.</p><p>We have 2 applications PingApp and PongApp and we have gateway application (Spring Cloud gateway), All work and make use of consul as and we have nginx as a reverse proxy. The imports are removed for brevity.</p><p>PingApp makes use of LoadBalanced RestTemplate with consul to communicate.</p><pre>@SpringBootApplication<br>@RestController<br>@EnableDiscoveryClient<br>class PingApp {<br><br>    @Autowired<br>    lateinit var restTemplate:RestTemplate<br><br>    @GetMapping(&quot;/&quot;)<br>    fun ping() = Message(&quot;pong-service -&gt; ${restTemplate.getForObject&lt;Message&gt;(&quot;http://pong-service/&quot;)}&quot;);<br>    @GetMapping(&quot;/self-ping&quot;)<br>    fun localPing() = Message(&quot;pong&quot;)<br><br>}<br>fun main() {<br>    runApplication&lt;PingApp&gt;()<br>}<br><br>data class Message(val message: String)<br><br>@Configuration<br>class RestConfig {<br>    @Bean<br>    @LoadBalanced<br>    fun restTemplate(): RestTemplate {<br>        return RestTemplate()<br>    }<br>}</pre><p>PongApp makes use FeignClient, I recommend this for all our own services and RestTemplate for simple one time external API call. Here FeignClient automatically discovers by the “ping-service” spring application name that’s registered with consul.</p><pre>@SpringBootApplication<br>@RestController<br>@EnableDiscoveryClient<br>@EnableFeignClients<br>class PongApp {<br><br>    @Autowired<br>    private lateinit var pingService: PingServiceClient<br><br>    @GetMapping(&quot;/&quot;)<br>    fun localPong() = Message(&quot;pong&quot;)<br><br>    @GetMapping(&quot;/ping-pong&quot;)<br>    fun remotePing() = Message(&quot;ping-service -&gt; ${pingService.ping()}&quot;)<br>}<br><br>fun main() {<br>    runApplication&lt;PongApp&gt;()<br>}<br><br>data class Message(val message: String)<br><br>@FeignClient(name = &quot;ping-service&quot;)<br>interface PingServiceClient {<br>    @GetMapping(&quot;/&quot;)<br>    fun ping(): Message<br>}</pre><p>This is how job spec for nomad looks like</p><pre>job &quot;ping-service-job&quot; {<br>  datacenters = [&quot;dc1&quot;]<br><br>  group &quot;ping-service-group&quot; {<br>    task &quot;ping-service&quot; {<br>      driver = &quot;raw_exec&quot;<br><br>      config {<br>        command = &quot;/bin/bash&quot;<br>        args = [&quot;-c&quot;, &lt;&lt;EOT<br>          # download from jenkins/ s3 jar url, Here I am doing from locally<br>          if [ ! -f /Users/pmanvi/nomad-experiments/ping-service/build/libs/ping-service.jar ]; then<br>            echo &quot;Error: ping service jar file not found!&quot;<br>            exit 1<br>          fi<br>          /Users/pmanvi/Library/Java/JavaVirtualMachines/openjdk-22.0.2/Contents/Home/bin/java  -jar /Users/pmanvi/nomad-experiments/ping-service/build/libs/ping-service.jar<br>        EOT<br>        ]<br>      }<br>      resources {<br>        cpu    = 500<br>        memory = 512<br>      }<br>    }<br>  }<br>}</pre><p>Gateway application.yaml, We have <a href="https://spring.io/projects/spring-cloud-gateway">great java dsl as well </a>and nginx reverse proxy for spring cloud gateway application and the how the output will look like this.</p><pre>server:<br>  port: 4001<br>spring:<br>  application:<br>    name: gateway-service<br>  cloud:<br>    gateway:<br>      discovery:<br>        locator:<br>          enabled: true<br>      routes:<br>        - id: ping-service<br>          uri: lb://ping-service<br>          predicates:<br>            - Path=/ping/**<br>          filters:<br>             - RewritePath=/ping/(?&lt;remaining&gt;.*), /${remaining}<br>        - id: pong-service<br>          uri: lb://pong-service<br>          predicates:<br>            - Path=/pong/**<br>          filters:<br>            - RewritePath=/pong/(?&lt;remaining&gt;.*), /${remaining}<br>    consul:<br>      host: localhost<br>      port: 8500<br>  config:<br>    import: optional:consul:localhost:8500</pre><pre>➜  ~ cat /opt/homebrew/etc/nginx/nginx.conf<br>worker_processes  1;<br><br>events {<br>    worker_connections  1024;<br>}<br><br>upstream springboot_gateway {<br>    server 127.0.0.1:4001;  # First instance of gateway, well we have consul template available with ngnix to use as well.https://github.com/hashicorp/consul-template<br>}<br><br><br>http {<br>    include       mime.types;<br>    default_type  application/octet-stream;<br><br><br>    sendfile        on;<br><br>    keepalive_timeout  65;<br><br><br>    server {<br>        listen 80;  # Listen on port 80 for HTTP requests<br>        server_name localhost;  # Replace with your domain<br><br>       location / {<br>          proxy_set_header Host $host;<br>          proxy_set_header X-Real-IP $remote_addr;<br>          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br>          proxy_set_header X-Forwarded-Proto $scheme;<br><br>          # Pass requests to the Gateway<br>          proxy_pass http://springboot_gateway;<br>       }<br>}<br><br>}</pre><pre>➜  ~ http localhost/ping/<br>HTTP/1.1 200 OK<br>Connection: keep-alive<br>Content-Type: application/json<br>Date: Sun, 13 Oct 2024 05:07:05 GMT<br>Server: nginx/1.27.2<br>Transfer-Encoding: chunked<br><br>{<br>    &quot;message&quot;: &quot;pong-service -&gt; Message(message=pong)&quot;<br>}<br><br><br>➜  ~ http localhost/ping/self-ping<br>HTTP/1.1 200 OK<br>Connection: keep-alive<br>Content-Type: application/json<br>Date: Sun, 13 Oct 2024 05:07:32 GMT<br>Server: nginx/1.27.2<br>Transfer-Encoding: chunked<br><br>{<br>    &quot;message&quot;: &quot;pong&quot;<br>}<br><br><br>➜  ~ http localhost/pong/<br>HTTP/1.1 200 OK<br>Connection: keep-alive<br>Content-Type: application/json<br>Date: Sun, 13 Oct 2024 05:07:41 GMT<br>Server: nginx/1.27.2<br>Transfer-Encoding: chunked<br><br>{<br>    &quot;message&quot;: &quot;pong&quot;<br>}<br><br><br>➜  ~ http localhost/pong/ping-pong<br>HTTP/1.1 200 OK<br>Connection: keep-alive<br>Content-Type: application/json<br>Date: Sun, 13 Oct 2024 05:07:54 GMT<br>Server: nginx/1.27.2<br>Transfer-Encoding: chunked<br><br>{<br>    &quot;message&quot;: &quot;ping-service -&gt; Message(message=pong-service -&gt; Message(message=pong))&quot;<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qp989Z8-5_g5Z4nxzq_WjA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UUYxVJUQbzjBoCKw3IImwA.png" /><figcaption>Here is how the nomad job UI and consul will look like.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wxABFFJTTO77661JZoAlsA.png" /><figcaption>Typical flow diagram with this setup</figcaption></figure><h4>What we (JVM, Spring shop) miss of k8s in this setup?</h4><p>Not much actually, k8s offers end-to-end container orchestration services including load balancing, configuration management, routing, feature gates, and service discovery. As we would like to use Spring’s features for all these that being native java, we don’t get benefit from these. Nomad is primarily a task scheduling platform that can orchestrate different workload types as an extended feature.</p><p>Of course the K8 has normalized<strong> , Automation and Self-Healing</strong> ,<strong>Portability and Flexibility</strong> , <strong>Service Discovery and Load Balancing,Storage and Configuration Management, </strong>but<strong> </strong>“don’t feel the need to rush to adopt Kubernetes, or even containers for that matter. They absolutely offer significant advantages over more traditional deployment techniques, but their adoption is difficult to justify if you have only a few microservices”</p><p>k8s is undoubtably king in this space and has more developers, maturity with unmatched developer and cloud eco system (helm charts, training material, engineers).</p><p>IMO, With Spring/JVM only eco system we don’t need pay the runway cost and accidental complexity it brings in because of feature list. (multi-cloud, multi-language support)</p><ul><li><strong>Features Missing in Spring Cloud</strong>: Auto-scaling, fault recovery, job management, deployment and scheduling, resource isolation, centralized telemetry.</li><li><strong>How Nomad Augments Spring Cloud</strong>: Nomad fills gaps by providing auto-scaling, fault tolerance, centralized telemetry, native job orchestration, and seamless resource management. It complements Spring Cloud’s strengths while offering lightweight alternatives to Kubernetes features. I</li></ul><p>Good for mixed workload environments</p><h4>Conclusion</h4><p>Nomad is known for its ease of setup and low learning curve, Nomad’s simpler scheduling system avoids the excessive abstraction and operational complexity found in Kubernetes</p><p>For lightweight Spring Boot services<strong> </strong>built on Spring Cloud, using Nomad for workload orchestration is efficient and resource-conscious choice. Without the overhead of Docker, we can focus on the simplicity and frugality of provisioning and orchestrating numerous Java services, while still benefiting from the robustness of Spring Cloud’s powerful discovery, tracing, and observability capabilities.</p><p>It may look odd to compare Spring Cloud and K8, but it does make sense as we need to pick which option makes sense for us. Nomad is non-invasive and allows us to use our favourite Spring features without worrying about “Monitoring and Logging”, “Load Balancing”, “Service Discovery” (spring works best with consul), “Configuration Management”, “Application Security” , “Gateway”… without guilt.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EtNnrDdXdJ_JlWyp.png" /><figcaption>From Hashicorp site.</figcaption></figure><p>Using http for internal services and htpps for external communication and implications over here.</p><p><strong><em>Isolated Network:</em></strong><br>If the private subnet is strictly isolated (no VPN, bastion, or direct access from outside), HTTP is tolerable for communication.<br><strong><em>Minimal OR No Risk of Insider Threats:</em></strong><br>If your environment has strong access control mechanisms (e.g., IAM, security groups, strict firewall rules), the risk of HTTP communication is lower to nil. Making sure that No HTTP communication exposes it to interception within the subnet.</p><p>Using HTTP in a tightly controlled, isolated environment is often acceptable, especially if the network is entirely within a single private, isolated subnet with no external access. It simplifies operations, reduces CPU overhead, and can be a reasonable tradeoff.</p><p>The private network is fully isolated. Security controls (SGs, NACLs) and monitoring are enforced, One can reassess the need for encryption if the environment evolves OR if some paranoid comes and insists on imaginary zero trust policies and other fancy words. For fully isolated internal network scenario, the current approach is pragmatic. The key is maintaining strict network isolation and being intentional about your security choices.</p><p>Rethink if,</p><ul><li>The network becomes more complex</li><li>Any external connectivity is introduced</li><li>Compliance requirements change</li><li>Security audit recommends against it</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3a4496f131ce5a9f9aaf308e7fbd3e4a/href">https://medium.com/media/3a4496f131ce5a9f9aaf308e7fbd3e4a/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fc315aafe29b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Advanced Data Security: Scope, Touch-less Design, and Implementation Strategies]]></title>
            <link>https://pmanvi.medium.com/advanced-data-security-scope-touch-less-design-and-implementation-strategies-fd7b79aa8ac6?source=rss-9398dbbcf83a------2</link>
            <guid isPermaLink="false">https://medium.com/p/fd7b79aa8ac6</guid>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[Praveen Manvi]]></dc:creator>
            <pubDate>Sun, 19 May 2024 14:17:08 GMT</pubDate>
            <atom:updated>2025-05-23T06:50:36.666Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Opt-In Security Architecture for Relational Databases with Automated Detection, Breach notification and Compliance Enforcement.</blockquote><h4>Overview</h4><p>Data security is often viewed unfavourably by developers and users, much like taxes. However, drawing from the wisdom of Chanakya’s taxation principles, the goal of security measures should be to fortify the digital environment, ensuring public welfare, safety, and sustainable development. Chanakya eloquently described the ideal approach: “<strong><em>Collect taxes from the citizens as honeybees collect nectar from flowers — gently and without causing harm.</em></strong>” This principle can be applied to data security and observability, emphasising the need for effective, unobtrusive methods. Here I am explaining a method that can result in zero changes to existing system (code and deployment) following plugin architecture principles.</p><p>Most organisations already have robust role-based systems that define CRUD operations at the entity level. However, these systems often do not support on-demand control of data, such as tagging arbitrary users or entities to control access to specific data rows within a set of entities.</p><p>Supporting field-level and row-level security is generally challenging because these cases are rare (assuming 1% of use cases) and can complicate the established role/group hierarchy that has evolved over time. This document tries to come up with an approach that aims to provide a simple solution without disturbing the existing system.</p><p>The idea is to develop a non-intrusive solution for implementing on-demand security for Row Level Security (RLS) and Field Level Security (FLS). For RLS, The core concept is to prevent access to specific data rows based on a policy, which can be arbitrary. The assumption is that this is a special case and falls into a small percentage of use cases. The goal is to implement this without disrupting the existing system (Users, Roles, Groups, Permissions), with the data residing in an RDBMS.</p><h4>Use cases</h4><p>Let me start with 2 requirements that needs to work with existing system and its “explicit include” and by default its include all.</p><p>We have ‘<strong>SalaryDetails</strong>’ entity that has ‘salary’ column needs to be hidden by all the queries except owner and few departments (Finance, Executives, HR) and restricted to few users among them as well. So Role level security will not help over here. Many a time these are transient set of users that will be permitted only for some time.</p><p>We have ‘<strong>ScanReportDetails</strong>’ entity that has ‘report’ (or say some of the ePHI fields) that needs to be encrypted and should be visible to arbitrary set of medical professionals that has consent from owner of the document (patient in this case). Many a time document uploaded by not the owner but some one of hospitals hence it becomes important to allow dynamic set of users to access.</p><p>Above use cases won’t fit into standard CRUD based standard ROLE, GROUP, ACLs. Its generally complicated implement hierarchy and allowing these kind of use cases. Its not worth to complicate implementing these special requirements.</p><p>Let’s also make sure from whom we are trying save this information from and implications. The Developers, System Admins, DevOps and any user who knows the primary key and can retrieve the data. Please note that it can be accidental and un-intentional updates happening from DevOps who generally will have access to databases without going through the databases. Of course we also want to save from applications accesing these content.</p><p>It’s security issues are ‘P0&#39; issues all the time as implications can be huge with data leakage. Having quick ability to recover from this will be great handle and this approach helps make it fast.</p><p><em>Customer identity theft results in financial losses/credit score damage etc…<br>Loss of trust and reputation of the enterprise in event of data breach and exposure.<br>Legal liability to the enterprise managing the PII Data<br>Export, Data pipeline, Caching Service and Reporting Service accessing this data.</em></p><h4>Design</h4><p>The outline of encryption process is depicted in below diagram</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OQCLwYRnqNISIZKAaeYA0A.png" /><figcaption>Encrypting Fields having sensitive information</figcaption></figure><p>Bring your own key (BYOK) functionality, Hashicorp is good solution with managing allowing customers to provide their own encryption keys for enhanced security and also integrates well with other Key Management Systems (KMS). <a href="https://developers.google.com/tink">Google Tink </a>provides field-level encryption using industry-standard AES-256 encryption with symmetric.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IghuOXw33Nz-igAD229TrQ.png" /><figcaption>Data flow diagram identifying the scope.</figcaption></figure><p>The “<a href="https://pmanvi.medium.com/touch-less-audit-trail-enabling-audit-and-building-data-traction-analytics-802249bd9419">touch less auditing architecture</a>” that I presented in earlier blog can augment this very well audit and compliance requirement in a transparent manner. “Encryption” and “Redaction” are approaches to implement the depending on the requirement although I provided the encryption redaction can be used in the same fashion. Following example provides how to use the library.</p><p><a href="https://gist.github.com/pmanvi/45b32d1d34392e9990104bf830a0ee86">https://gist.github.com/pmanvi/45b32d1d34392e9990104bf830a0ee86</a></p><pre>import com.google.crypto.tink.Aead<br>import com.google.crypto.tink.KeysetHandle<br>import com.google.crypto.tink.aead.AeadConfig<br>import com.google.crypto.tink.aead.PredefinedAeadParameters<br>import com.nerdvision.redact.pii.PiiRedactor<br><br>fun main() {<br>    val input = &quot;pii data like aadhar number email(pmanvi@outlook.com), phone ( 9xxxx295 )&quot;<br><br>    val redactedText = PiiRedactor.getInstance().redact(input);<br>    println(redactedText.contains(&quot;EMAIL_ADDRESS&quot;))<br>    println(redactedText)<br>    // pii data like aadhar number email(EMAIL_ADDRESS), phone ( PHONE_NUMBER )<br><br>    AeadConfig.register();<br>    // simple and robust symmetric encryption (AES) that employs a single key to<br>    // both encrypt and decrypt the data.<br>    val aad = &quot;shared secret read from Hashicorp vault&quot;.toByteArray()<br><br>    val aead = KeysetHandle.generateNew(PredefinedAeadParameters.AES256_GCM).getPrimitive(Aead::class.java)<br><br>    println(&quot;Original: $input&quot;)<br>    val ciphertext = aead.encrypt(input.toByteArray(), aad);<br><br>    // Will be stored in database.<br>    println(&quot;Ciphertext: ${String(ciphertext)}&quot;)<br><br>    val original = String(aead.decrypt(ciphertext, aad))<br><br>    println(&quot;Decrypted: $original&quot;)<br><br>}<br><br><br>/**<br> * built in regex based redaction rules for:<br> * credentials<br> * creditCardNumber<br> * emailAddress<br> * ipAddress<br> * name<br> * phoneNumber<br> * streetAddress<br> * usSocialSecurityNumber<br> * zipcode<br> * url<br> * digits<br> */</pre><h4>Automated tagging of fields and controlling visibility</h4><p>If the system is already having 1000s of tables and its almost impossible and its not fruitful also go back and tag the fields. Even for new green field projects, expecting user to be technically/legally aware to tag field correctly is stretch ask &amp; when they look for ePHI and PII fields information available in internet which will too broader and makes system in-efficient, for PaaS provider its there responsibility to make sure that there is no security breach and depending on user to tag the columns will not be scalable option.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*buhrmBm8fboLpqbRy_phMg.png" /><figcaption>Using CDC and ML libraries to mark/encrypt the sensitive fields.</figcaption></figure><h4>Row Level Security By Group</h4><p>It really helps to have separate concept as ‘Row Level Security Group’ dedicated for a managing such use cases. <em>Please note the caveat that the whole assumption is that its just 1% use cases and we do not want to complicate existing CRUD Role, ACLs for this. If this is the main use case, I don’t think this approach should not be considered.</em></p><p>One of the reason I did this POC/experiment was to evaluate ‘<a href="https://jetbrains.github.io/Exposed/home.html">exposed</a>’ ORM framework from Kotlin and used SpringBoot as framework. xposed is an open-source library (Apache license) developed by JetBrains, which provides an idiomatic Kotlin API. I don’t think I can recommend for large scale applications. It was from brilliant JetBrains engineer. I will have separate write up on the same.</p><p>Picking of user sets for RlSGroup and and entity can be driven by Rules Engine based ‘User’ and ‘Entity’ supported by rule engine editor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ueQQI7zS_D5TSQ3XOgvfhw.png" /><figcaption>Database Entities and relationships, Here User/Entity tables are considered existing ones</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rd0e-_z04qLlER9nu9-zvg.png" /><figcaption>Kotlin Data class representation of schema</figcaption></figure><h4>Sample Data and Queries</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WxfMOF0Z8pdSV6KxzBYR7w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aazbZABtCGGRMn2YhelMdw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z6ZFQexzITJYkSWe9htyFQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*5BqBTZZzaUh-dAxrDNpLUQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lHyJhovP9B-UlWg1IeZ_oA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9-9EevAgWG3FGQJ6J5ZLNg.png" /><figcaption>Sample Data of Entities and and Row Level Security Groups</figcaption></figure><pre>SET @TEST_USER_ID = 5;<br>SET @TEST_GROUP_ID = NULL;<br><br>SELECT entity.*<br>FROM ScanReportDetails entity<br>         JOIN Users u ON entity.ownerId = u.id<br>         LEFT JOIN (<br>    SELECT userId, 1 AS groupId<br>    FROM UserRlsGroups<br>    WHERE groupId = @TEST_GROUP_ID<br>    UNION ALL<br>    SELECT entityId AS userId, 1 AS groupId<br>    FROM EntityRlsGroups<br>    WHERE entityType = &#39;ScanReportDetails&#39; AND groupId = @TEST_GROUP_ID<br>) ug ON u.id = ug.userId<br>WHERE ug.userId IS NOT NULL OR u.id = @TEST_USER_ID;</pre><p>For userId 5 with group_id (NULL), he will see only his record as he owner, but userId 1 &amp; 2 the rls_group_id is 1 they will see records of other docs as well as their group_id has access to. This query is optimised thinking most of the users will have NULL as group_id without relying on ‘IN subquery’</p><p>HIPAA mandates specific security standards for protecting ePHI, including requirements for access controls, encryption, audit controls, and safeguards against un-authorised access or disclosure. Compliance with HIPAA requires adherence to the HIPAA Security Rule, Privacy Rule, and Breach Notification Rule. Under HIPAA, certain information about a person’s health or health care services is classified as Protected Health Information (PHI)</p><p><a href="https://huggingface.co/spaces/presidio/presidio_demo">Presidio Demo - a Hugging Face Space by presidio</a></p><p><a href="https://github.com/microsoft/presidio">https://github.com/microsoft/presidio</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fd7b79aa8ac6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Touchless Audit Trail: Enabling Audit and Building Data Traction Analytics]]></title>
            <link>https://pmanvi.medium.com/touch-less-audit-trail-enabling-audit-and-building-data-traction-analytics-802249bd9419?source=rss-9398dbbcf83a------2</link>
            <guid isPermaLink="false">https://medium.com/p/802249bd9419</guid>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Praveen Manvi]]></dc:creator>
            <pubDate>Fri, 29 Mar 2024 06:53:04 GMT</pubDate>
            <atom:updated>2025-04-06T16:01:31.654Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Revolutionising Data Auditing with a non-invasive framework</blockquote><h3>Overview</h3><p>The “Touch-less Audit Trail” system is designed to enable the auditing of primary source data without requiring any code changes to existing systems. This innovative solution leverages a plugin Change Data Capture (CDC) architecture, ensuring seamless integration and minimal OR no disruption to existing processes. The ability to make sense and react upon data changes in real-time is a critical differentiator.</p><h3>Design</h3><p>In the realm of data management, ensuring the integrity and accountability of primary source data is paramount. However, traditional auditing methods often entail cumbersome code modifications and system disruptions, posing significant challenges for organizations striving to maintain operational efficiency while meeting regulatory standards. At its core, the “Touch-less Audit Trail” system is built upon a sophisticated architecture that seamlessly integrates with primary data sources, leveraging Change Data Capture (CDC) technology to capture data changes in real time.</p><h3>Design Components</h3><h4>Architecture</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/759/1*UTdByDMiR_UHRsUTgunQTw.png" /></figure><ol><li>Debezium: Acting as a powerful source connector, Debezium captures change data directly from primary data stores such as MongoDB or PostgreSQL. By streaming change events to Apache Kafka topics, Debezium preserves data integrity while obviating the need for disruptive modifications to the source system.</li><li>Javapoet/Java CodeGen Connector: This component plays a pivotal role in processing Data Definition Language (DDL) statements and generating corresponding Java code. By utilizing this generated code, organisations can track and audit data changes without altering their existing codebase, ensuring seamless integration with the “Touchless Audit Trail” system.</li><li>Javers 7.1: Leveraging the generated Java code, Javers facilitates the capture of object changes and commit history. This robust library enables comprehensive tracking of data modifications, empowering organizations to maintain a detailed audit trail without impeding the functionality of their source systems.</li><li>History Connector: Serving as the orchestrator of the commit process, the History Connector integrates the generated Java code, data changes from Kafka topics, and the Javers library. By facilitating the storage of change history and snapshots in the Entity Version Database (DB), this component establishes a centralized repository for audit and reconciliation purposes.</li><li>Entity Version DB: Dedicated to maintaining a comprehensive record of entity versions, commit history, and snapshots, the Entity Version DB serves as a cornerstone of the “Touchless Audit Trail” system. This centralized repository enables efficient querying and analysis of historical data, supporting various audit, debugging, and reconciliation efforts.</li><li>History-service: Acting as a dedicated service component, the History-service interfaces with the Entity Version DB, allowing controlled access to historical data through Javers DSL queries. This gateway provides organizations with the means to retrieve commit history and snapshots for auditing, debugging, and reconciliation purposes.</li></ol><p>Many a time for business applications it makes sense to view an audit from the parent table in a relational database. The solution is to maintain the parent-child relationship in a config and bump up the version of the parent object with requirements changed child references. This gives seamless experience to see the lineage and look at the history change from a business perspective.</p><p>It&#39;s important to know the traction of entity changes, say by the author. Since we have CDC available, we can build metrics by user, table, and customer to build a heat map on traction analytics.</p><p>Generated code provides an option to implement factory methods to look up the data that can be used in internal SDKs in a type type-safe manner.</p><h3>Summary</h3><p>By implementing the “Touch-less Audit Trail” system, organizations can achieve comprehensive auditing of their primary source data without disrupting existing systems or modifying critical codebases. This approach minimizes risks, reduces development efforts, and ensures compliance with data governance and regulatory requirements while maintaining operational efficiency and integrity.</p><h4>Why and how about Javers library?</h4><p>At the heart of Javers lies its core functionality, which includes tracking changes, managing snapshots, and providing querying capabilities to retrieve historical data. Javers uses a combination of reflection and annotations to track changes in Java objects, javers captures any mutations and stores them in its repository that is friendly to query the versions.</p><p>When developing applications, we often encounter the need to store information about how data has changed over time. This information can be used both to debug the application more easily and to meet design requirements. In this article, I will discuss an implementation pattern to implement an audit trail with the JaVers library, which allows us to automate this process by recording changes in the states of database entities.</p><p>Why Mongo:<br>MongoDB’s horizontal scalability makes it capable of efficiently storing and retrieving versioned data for thousands of entities. Given the nature of query patterns and storage semantics, with the experiments it was found it’s more efficient in terms of storage and query compared to RDBMS (Postgress).</p><p>For Mongo,<br>jv_head_id — Collection for storing the document containing the last value of commitId<br>jv_snapshots — This collection contains detailed information about changes made to a business entity as a result of a creation, update or delete operation.</p><p>Queries:<br>Shadow — contains the historical data of the domain object, which are reconstructed recreated from snapshots (snapshots)<br>Change — represents the difference between the properties of two objects<br>Snapshot — contains historical data of a domain object, represented as a property map with values</p><p>Changed properties are maintained in a manner that is consistent to compare any of version with any. exactly working like git history. Its a framework designed for object auditing and diffing in Java, stores only the changed data rather than the entire data for newer versions. This approach is in line with its purpose to provide efficient and focused auditing capabilities. By storing only the changes rather than the entire object state each time, JaVers ensures efficient use of storage and makes it easier to track and review modifications.</p><p>In summary, JaVers focuses on storing incremental changes (diffs) between versions of an object, which optimizes storage and provides clear auditing trails for modifications.</p><p>Scaling MongoDB</p><p>The query pattern and the use cases as queries will be particular schema or tenant. It&#39;s easy to scale this instance both vertically and horizontally (well supported by most of the cloud vendors). Depending on the compliance and reporting requirements.</p><h4>References</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1IbNMkV3zkuK8f3qe9bN3A.png" /></figure><ul><li><a href="https://javers.org/">JaVers - The Leading Framework for Object Audit and Diff in Java</a></li><li><a href="https://docs.confluent.io/platform/current/connect/kafka_connectors.html">Kafka Connectors | Confluent Documentation</a></li><li><a href="https://developer.squareup.com/blog/introducing-javapoet/">Introducing JavaPoet</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=802249bd9419" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[REST API Guidelines]]></title>
            <link>https://pmanvi.medium.com/rest-api-guidelines-78c9e51056b4?source=rss-9398dbbcf83a------2</link>
            <guid isPermaLink="false">https://medium.com/p/78c9e51056b4</guid>
            <dc:creator><![CDATA[Praveen Manvi]]></dc:creator>
            <pubDate>Sun, 25 Feb 2024 17:14:47 GMT</pubDate>
            <atom:updated>2025-01-06T12:39:29.758Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>“Easy to read, extend, hard to misuse, memorize and more importantly focusses on single responsibility does efficiently and more most importantly domain friendly” == “Good API”</blockquote><p>UX can either make or break a business and in today’s growing API Economy, it is the Developer Experience (DX) that holds the same weight. Companies like Google, Meta, and Netflix, part of the API billionaire’s club, have embraced an API-first development approach. This strategy prioritizes the creation of APIs tailored to meet the needs of developers before constructing the final product, whether it’s a website, mobile app, or SaaS software.</p><p>Even if businesses opt not to expose their APIs publicly, it’s crucial to recognize their significance. Neglecting APIs can indicate a lack of foresight. As Forrester aptly puts it, APIs serve as digital glue, fostering the coexistence of diverse technologies, opening doors to new business opportunities, accelerating time-to-market, and enabling the flexible manipulation of data for various visualisations to adapt swiftly to emerging trends.</p><p>Embracing the philosophy of “eating your own API” for even internal applications can also reap the benefits of well-designed APIs. Amazon exemplifies this approach with its emphasis on services and interfaces, demonstrating how internal systems can thrive by leveraging robust APIs.</p><p><strong><em>Speed</em></strong>:<br>Aim for optimal speed, ensuring it meets requirements without compromising quality.<br><strong><em>Security</em></strong>:<br>Prioritize absolute security, leaving no room for vulnerabilities. “Good enough” is not acceptable in this domain.<br><strong><em>Learnability</em></strong> and Extensibility:<br>Ensure ease of learning and the ability to extend functionalities seamlessly.<br>Design with the user in mind, making it easy to use while minimizing the potential for misuse. Tailor appropriateness to the target audience.<br><strong><em>Discoverability</em></strong>:<br>Make the system easily discoverable, allowing users to find what they need intuitively.<br><strong><em>Loose Coupling:</em></strong><br>Foster loose coupling between components, enhancing discoverability and promoting flexibility in integration.</p><p><a href="https://lawsofux.com/postels-law/">Postel’s Law</a> encapsulates the essence of good API</p><p>while dealing with requests, Be stringent in what is sent out, maintaining control and awareness of consumer interactions. Prevent any unintended information leakage. Be flexible in what is accepted, allowing providers to include additional information without disrupting consumer functionality.<br>When defining rules, lean towards strictness initially to minimize disruptions. It’s easier to loosen restrictions later than to tighten them retrospectively.</p><p>Define API’s outside code, so it can be reviewed by stakeholders. Get feedback with Specification using OpenAPI specification (Swagger Specification) API endpoints must be secure and backward compatible,<br><em>JSON Guidelines<br></em>APIs must always return JSON objects as a Top level data structure to support extensibility Array names should be pluralized<br>Boolean properties must not be null<br>Enumerations should be represented as Strings<br>Date property values should conform to <a href="https://datatracker.ietf.org/doc/html/rfc3339#section-5.6">RFC 3339</a><br>Time duration and interval could conform to <a href="https://en.wikipedia.org/wiki/ISO_8601">ISO 8601</a></p><h4>RESTful URLs and actions</h4><p>Endpoint names (Use plural nouns and concrete names). Avoid having verbs in rest endpoints.<br>Use nouns /exams ☑️ but not /getExams<br>Use plural /exams ☑️ but not /exam <br>Use concrete names /cars☑️ but not /vehicles<br>Simplify associations (Limit resource levels to be not more than 2 nested resources)<br>Resources almost always have a relationship with other resources. Do not include too many levels. Once the primary key for one level exists, others can be derived /employees/101/tasks ☑️ and not /orgs/1/employees/101/tasks</p><p>While developing spring boot applications, make sure to have dedicated ‘rest’ controller (/api end point) focussing on developer experience and not clubbed with ui serving controller (/ui controller serving say thymeleaf pages)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*9QE1Yd10yE00nedd6EIY2A.png" /><figcaption>Example of Rest resources and recommendation for packages</figcaption></figure><h4><strong><em>Versioning</em></strong></h4><p>There is strong argument that by default we should always work without versions and newer features can be made with backward compatibility, but I thnk it’s better to make API versioning is mandatory as we always live with multiple implementation and will go through maturity phase.<br> Specify the version with a ‘v’ prefix. Move it all the way to the left in the URL so that it has the highest scope. Version numbers should be a simple ordinal number Ex: <strong><em>/v1/exams</em></strong></p><h4>API Key</h4><p>As APIs have to go through gateway, then API Key should be the path segment next to version. It makes much easier for routing rules. Its also can be kept as header, but I think given caching, routing, ratelimiting, pricing works better with path param in the long run. Ex: <strong><em>/v1/{apiKey}/exams</em></strong></p><h4>Naming Conventions &amp; End Point Guidelines</h4><p>As camelCase is standard java symantic, it may be better to stick this naming convention for all parameter names.<br>Ex: <strong><em>/exams?fields=name,dateTime,lastUpdatedDateTime</em></strong></p><p>Use a unique query parameter for each field that implements filtering Ex: <strong><em>/exams?author=pranav</em></strong></p><p>generic parameter sort can be used to describe sorting rules. Use limit and offset as parameters<br>Ex: <strong><em>/exams?limit=25&amp;offset=50</em></strong></p><p><em>Updates &amp; creation should return a resource representation</em><br>A PUT, POST or PATCH call may modify fields of the underlying resource that weren’t part of the provided parameters. API should return the updated (or created) representation as part of the response. For e.g., In case of a POST that resulted in a creation, HTTP 201 status code should be returned and include a Location header that points to the URL of the new resource.</p><h4>Path Params Vs Query Params</h4><p>Unique identifiers of the requested resource should be in a path segment rather than a query parameter. Optional parameters can be part of query parameters. Query parameters can be used for filtering or sorting.</p><h4>Error Management</h4><p>An API should provide a useful error message in a known consumable format. It must return HTTP status codes. Define a consumable error payload for 400 series errors. The API should standardize that all 400 series errors come with consumable JSON error representation</p><pre> {<br>      &quot;userMessage&quot;: &quot;Sorry, the requested resource does not exist&quot;,<br>      &quot;internalMessage&quot;: &quot;No exam found&quot;,<br>      &quot;code&quot;: &quot;2002&quot;<br>   }</pre><h4><strong><em>Performance &amp; Security</em></strong></h4><p>Only include required information end users request. Partial result is good idea that graphql kind of interface with REST is useful that product like elasticsearch exploits pretty well with their RESTful APIs.<br>“fields” query parameter should take a comma-separated list of fields to include Ex: <strong><em>/examps?fields=name,author</em></strong></p><p><em>Must support Pagination for anything that comes as list. Limit the records returned to not more than 100 to avoid large payloads.</em></p><p><em>GET requests should not disclose sensitive information as it can get bookmarked</em></p><p><em>SSL everywhere, all the time</em></p><h4>Selecting HTTP status code</h4><p>The selection of HTTP status code is important. Following guidelines should be easy to remember and pick. Its also important to understand when we deal with external end point integration. Here is overview look at the details. My suggestion would be stick to 200 series for everything and rely on detailed message for managing alternate flows.</p><p>Informational responses (100–199) — Don’t worry, you hardly need to worry about.<br>Successful responses (200–299) — “Sab changasi” — Server is saying all is well with request and server.<br>Redirection messages (300–399) — “Server is saying I am asking help with some other server”<br>Client error responses (400–499) — Server saying to client, you screwed up check your request.<br>Server error responses (500–599) — Server saying “Sorry, Forgive me something I screwed up”</p><p>There are no standards here Taking example for Meta, No matter what happens on a Facebook request, you get back the 200-status code — everything is OK. Many error messages also push down into the HTTP response. I felt this is better model that worrying about all the status code. Other companies do adher to status codes.<br>HTTP Status Code: 200<br>{“type” : “OauthException”, “message”:”(#803) Some of the<br>aliases you requested do not exist: foo.bar”} We can stick to these -,<br>200-OK, 400 — Bad Request, 500 — Internal Server Error, 201 — Created, 404 — Not Found &amp; 401 — Unauthorized</p><h4>Reference</h4><p>Scope of API domain and API as product.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/581/1*zu51DobRuxeiFWgbzQzciw.png" /></figure><p><a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Status">HTTP response status codes - HTTP | MDN</a></p><p>Examples to refer:</p><p>Stripe, a popular payment processing platform, offers a good examples for RESTful API with descriptive URLs like /v1/customers for managing customers and /v1/payments for handling payments. They use appropriate HTTP verbs such as POST for creating resources and PUT for updating them. <a href="https://docs.stripe.com/api">https://docs.stripe.com/api</a></p><p>Sample RestController class for reference.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e8453e6a387ef85a5d56063220bd19c7/href">https://medium.com/media/e8453e6a387ef85a5d56063220bd19c7/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=78c9e51056b4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stakeholder driven logs]]></title>
            <link>https://pmanvi.medium.com/stakeholder-driven-logs-42954da79635?source=rss-9398dbbcf83a------2</link>
            <guid isPermaLink="false">https://medium.com/p/42954da79635</guid>
            <category><![CDATA[spring-boot]]></category>
            <category><![CDATA[guidelines]]></category>
            <category><![CDATA[logging]]></category>
            <category><![CDATA[java8]]></category>
            <dc:creator><![CDATA[Praveen Manvi]]></dc:creator>
            <pubDate>Tue, 24 Oct 2023 04:44:14 GMT</pubDate>
            <atom:updated>2025-11-17T18:40:30.944Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Guidelines of logging in JVM Applications</blockquote><p>Whenever we set up a new project, one important activity is to set up a logging standard. In spite of APM, tracing tools advancement and observability tools, there is no match for well-designed logs for debugging. If the logging looks extra work, it&#39;s a symptom but not a problem. This document comes with standards that need to be followed for developing java applications and services. Thoughtful logging is a powerful mechanism that enables us to track events and detect potential bugs and just for debugging. In this blog post, we’ll explore the importance of logging. There are two goals of the debugging through logs, a. to uncover and to correct any misconceptions of the programmers; b. to improve the debugging abilities of the developers and QA engineers. Debugging with stakeholder-driven logs is an iterative process, and it’s essential to maintain effective communication with stakeholders throughout the cycle to ensure their needs are met and that the software aligns with their expectations.</p><p>As the log level is a fundamental concept in logging, no matter which logging framework you use, It is important that to include contextual information. In many companies that I have seen, looking at kibana/grafana will make me sad. All the stake, holders, including end-users, product managers, or customer support teams should be encouraged to go through these logs and come up with what makes sense. So much information is getting spitted out and most of them other adding to observability cost add little value. This document comes up with a recommendation on how to log from the stakeholder OR customer perspective.</p><p>Here are some basic general principles to start with.</p><p><em>Use the slf4j facade for logging.</em></p><pre>private static final Logger log = LoggerFactory.getLogger(ReadinessService.class);</pre><p>Please take a log at Loggers should be named for their enclosing classes, Please note it&#39;s not LOGGER (as it&#39;s not primitive although constant and looks not so good and also not ‘logger’ although noun and correct, log is simple and more readable given its usage every where.</p><p><em>Parameterise logging statements, make sure that the number of curly braces matches the number of parameters that were used. i.e. log.info(“Hello “ + user) becomes, Don’t have any method calls OR calculations inside the logs.</em></p><pre>log.info(&quot;Hello parameterized user {}&quot;, user).</pre><p>If exceptions are being handled as alternate flows (which is btw is highly discouraged to design, design exceptions as runtime and make sure they are not used as alternate paths), while logging these it should be logged as info and not as error OR warn, Fluent API setCause should help. It is a common mistake to call Exception.getMessage() when passing an exception into a log method. Not all exception types have useful messages, and even if the message is useful, this omits the stack trace. the error method has the Throwable written as a parameter value. Of course this should be refactored if we have the control.</p><pre>log.atInfo()<br> .setMessage(&quot;Checking the Exception Logging is running&quot;)<br> .addArgument(LocalDateTime.now())<br> .setCause(exceptionCause)<br> .log();</pre><p>&amp; of course never log security-sensitive information like passwords, PII (Personal Identifiable Information), servers … kind of data.</p><h3>Logging levels</h3><p>The following examples are explained with the rationale to help make informed decisions for picking logging levels. Levels should be considered from a stakeholder perspective.</p><p><em>level ( “targeted audience” )</em></p><blockquote>log.debug(<strong>support, enable optionally from users for diagnosing operational issues</strong>)</blockquote><p>Fine-grained information that is useful to programmers for investigation, P.S. complete details, please note that these logs will never be enabled in production, only on need basis a particular class can be enabled for debugging purpose.</p><pre>log.debug(&quot;Search Request {} &quot;, searchRequest);  <br>log.debug(&quot;Db trigger event {} &quot;, dbEvent); </pre><p>Micro performance consideration.</p><p>the penalty of invoking SLF4J in suppressed mode is in the order of 10 ns, which can be significant for highly performant code and tight loops, check logging rather than the having if conditions that impose a double-checking penalty if logging is enabled. Varargs are just syntactic sugar; the performance impact is that of array allocation. There is effort like <a href="https://github.com/obsidiandynamics/zerolog">zero logging</a> for low-overhead logging façade for performance-sensitive applications, but I don’t think it&#39;s applicable for spring boot applications, but it&#39;s good to be aware of overheads.</p><pre>private final boolean DEBUG = false;<br>if (DEBUG) {<br> log.info(&quot;Parse tree {} &quot; + tree.toStringTree());<br>}</pre><p>This can make sure that code will not get compiled into byte code as well.</p><blockquote>log.info(<strong>administrators for auditing</strong>)</blockquote><p>Changes such as configuration settings, entry/exit points for important processes like scheduled jobs running, services starting and stopping, and changes to data. Try to get the contextual information as much as possible and be descriptive.</p><pre> log.info(&quot;Created local tiered storage manager [{}]:[{}]&quot;, brokerId, storageDirectoryName);<br><br> if (snapshotId.offset() &lt; startOffset()) {<br>            log.info(<br>                &quot;Cannot create a snapshot with an id ({}) less than the log start offset ({})&quot;,<br>                snapshotId,<br>                startOffset());<br>   }</pre><blockquote>logger.warn(<strong>automated systems, operators, and administrators</strong>)</blockquote><p>Any condition that, while not an error in itself, may indicate that the system is running suboptimally. Warnings should highlight an abnormal or unexpected event in the application flow. This event does not cause the application execution to stop. invalid authentication, higher memory/disk usage, etc… Use a threshold to log time sensitive with saying &gt;200 ms etc along with absolute value, It helps to search from Kibana</p><pre>log.warn(&quot;Graceful shutdown timed out after &#39;{}&#39; ms&quot;, finishTimer.timeoutMs());<br>log.warn(&quot;Found unexpected directory {}. Will not delete.&quot;, notAFile.get().getAbsolutePath()));<br><br>// say 200 is SLA latency<br>if(timeTookInMillis &gt; 200) {<br>     log.warn(&quot;The request {} took &gt;200 ({} ms) tp process&quot;, request, timeTookInMillis);<br>}</pre><blockquote>log.error(<strong>monitoring systems and DevOps</strong>)</blockquote><p>This is the one that’s misused a lot. Any log with an error should result in some recover action, if not this should be refactored as warn as info. Any message with a condition that indicates something has gone wrong with the system is what the error stands for. Generally, all exceptions should be logged with the error, which also says not to use exceptions for logic but in exceptional cases only. All programmatic inconceivable conditions should also be logged as errors. Throw IllegalStateException in those cases after logging the context.</p><pre>log.error(&quot;Error occurred while reading the remote data for {}&quot;, fetchInfo.topicPartition, e);<br><br> // Its important to tag non exceptional error conditions that requires immendeate attention<br> if (partitionResponse.snapshotId().epoch() &lt; 0) {<br>                    logger.error(<br>                        &quot;The leader sent a snapshot id with a valid end offset {} but with an invalid epoch {}&quot;,<br>                        partitionResponse.snapshotId().endOffset(),<br>                        partitionResponse.snapshotId().epoch()<br>                    ); </pre><blockquote>log.severe(<strong>EveryOne</strong>)</blockquote><p>All hands on desk situation A critical issue that should describe an unrecoverable application or system crash, or a catastrophic failure that requires immediate attention. All integration and bootstrap errors should be logged at this level.</p><pre> // Usually all startup failure errors<br> log.severe(&quot;Unable to create connection from DataSource. Exiting ...&quot;);</pre><h4>Managing with Spring Applications</h4><p>As spring is a de-facto standard, here is a recommendation to be used as logger configuration. The pattern makes use of <a href="https://logback.qos.ch/manual/mdc.html">MDC</a> to log apiKey and user context information. It&#39;s invaluable in multi-tenant environments.</p><p>spring-logback-xml (for all spring boot services the resources directory can contain this file) , make sure to use the ‘spring.application.name’ in the application.properties file OR application.yaml file. use the &lt;servicename&gt; pattern. — logging.config=file:/opt/config/logback.xml</p><pre>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;<br>&lt;configuration&gt;<br>    &lt;!-- Spring Properties --&gt;<br>    &lt;springProperty scope=&quot;context&quot; name=&quot;service&quot; source=&quot;spring.application.name&quot; defaultValue=&quot;application&quot;/&gt;<br>    &lt;springProperty scope=&quot;context&quot; name=&quot;logLevel&quot; source=&quot;logging.level.root&quot; defaultValue=&quot;WARN&quot;/&gt;<br>    &lt;springProperty scope=&quot;context&quot; name=&quot;logPath&quot; source=&quot;logging.file.path&quot; defaultValue=&quot;./logs&quot;/&gt;<br>    <br>    &lt;!-- Console Appender with color --&gt;<br>    &lt;appender name=&quot;CONSOLE&quot; class=&quot;ch.qos.logback.core.ConsoleAppender&quot;&gt;<br>        &lt;encoder&gt;<br>            &lt;pattern&gt;%d{HH:mm:ss.SSS} [%highlight(%-5level)] %cyan(%logger{15}) - %msg%n&lt;/pattern&gt;<br>        &lt;/encoder&gt;<br>    &lt;/appender&gt;<br>    <br>    &lt;!-- File Appender --&gt;<br>    &lt;appender name=&quot;ROLLING&quot; class=&quot;ch.qos.logback.core.rolling.RollingFileAppender&quot;&gt;<br>        &lt;file&gt;${logPath}/${service}.log&lt;/file&gt;<br>        &lt;rollingPolicy class=&quot;ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy&quot;&gt;<br>            &lt;fileNamePattern&gt;${logPath}/${service}-%d{yyyy-MM-dd}.%i.log.gz&lt;/fileNamePattern&gt;<br>            &lt;maxFileSize&gt;10MB&lt;/maxFileSize&gt;<br>            &lt;maxHistory&gt;7&lt;/maxHistory&gt;<br>            &lt;totalSizeCap&gt;100MB&lt;/totalSizeCap&gt;<br>            &lt;cleanHistoryOnStart&gt;true&lt;/cleanHistoryOnStart&gt;<br>        &lt;/rollingPolicy&gt;<br>        &lt;encoder&gt;<br>            &lt;pattern&gt;%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n&lt;/pattern&gt;<br>        &lt;/encoder&gt;<br>    &lt;/appender&gt;<br>    <br>    &lt;!-- Async Appender for better performance --&gt;<br>    &lt;appender name=&quot;ASYNC_ROLLING&quot; class=&quot;ch.qos.logback.classic.AsyncAppender&quot;&gt;<br>        &lt;appender-ref ref=&quot;ROLLING&quot; /&gt;<br>        &lt;queueSize&gt;512&lt;/queueSize&gt;<br>        &lt;discardingThreshold&gt;0&lt;/discardingThreshold&gt;<br>        &lt;includeCallerData&gt;false&lt;/includeCallerData&gt;<br>    &lt;/appender&gt;<br><br>    &lt;logger name=&quot;org.springframework&quot; level=&quot;WARN&quot;/&gt;<br>    &lt;logger name=&quot;org.hibernate&quot; level=&quot;WARN&quot;/&gt;<br>    <br>    &lt;!-- Root logger --&gt;<br>    &lt;root level=&quot;${logLevel}&quot;&gt;<br>        &lt;appender-ref ref=&quot;ASYNC_ROLLING&quot; /&gt;<br>        &lt;!-- &lt;appender-ref ref=&quot;CONSOLE&quot; /&gt; --&gt;<br>    &lt;/root&gt;<br>&lt;/configuration&gt;</pre><h4>Summary</h4><p>Hope this provides some general sense and acts as checklist for code review and enhance our ability effectively make use of logs. Be judicious about the information that we log, it’s a database and costs money to store and for faster retrieval through full text search. Avoid logging obvious stuff and duplicating the information.</p><h4>Reference</h4><p><a href="https://google.github.io/styleguide/javaguide.html">https://google.github.io/styleguide/javaguide.html</a> <a href="https://www.oracle.com/java/technologies/javase/seccodeguide.html">https://www.oracle.com/java/technologies/javase/seccodeguide.html</a></p><p>Some of the examples are taken from popular open source frameworks such as Kafka and Elastic.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42954da79635" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Rule Engines, Scope and Patterns]]></title>
            <link>https://pmanvi.medium.com/rule-engines-scope-and-patterns-b6b7b4737f38?source=rss-9398dbbcf83a------2</link>
            <guid isPermaLink="false">https://medium.com/p/b6b7b4737f38</guid>
            <dc:creator><![CDATA[Praveen Manvi]]></dc:creator>
            <pubDate>Sun, 10 Sep 2023 11:39:13 GMT</pubDate>
            <atom:updated>2023-10-02T13:14:45.711Z</atom:updated>
            <content:encoded><![CDATA[<p>This writeup discusses about the scope of Rule Engine (Drools) and patterns, anti patterns of its usage.</p><h3>What is Rule Engine?</h3><p>“<em>Drools is a business rule management system with a forward-chaining and backward-chaining inference based rules engine, allowing fast and reliable evaluation of business rules and complex event processing. A rule engine is also a fundamental building block to create an expert system which, in artificial intelligence, is a computer system that emulates the decision-making ability of a human expert</em>.” as defined in their <a href="https://github.com/apache/incubator-kie-drools">github</a> project. Update (Congrats to Drools for becoming top level apache project as I heard on Sept 16th 23)</p><p>Tactical/Operational rules/decisions greatly influence bottom-line profitability. Aligning operational decisions with strategic ones if we rules to be modified quickly and made functional.<br>business rule is a statement that conforms to a when/then type construct,i.e. conditions/consequence.<br>Ex:<br>Decide Free shipping for customer — (premium, location, price, new user, loyalty)<br>Place ads for user from buying history and other attributes.<br>For new employess send invitation for medical insurence plan with new updated change rules.<br>React differently based customer (premium) attributes complaints.</p><p>Some design decisions are point in time and can’t be thought at the design time &amp; context specific, having this ability to change rules (operations) and just reference values (price, age, location, time, and similar attributes) participating in decisions can result in huge operational benefits. Replicating Rule Engine (Ex:Drools) abilities using conventional coding techniques is challenging and expensive. Centralized rules can help legal compliance and dynamic change control &amp; can be easily preserved and used to store and upsell as domain experise.</p><h3>Why Rule Engine?</h3><h4>Dealing with Complexity of Decision-Making</h4><p>It allows you to define rules/business logic in a declarative way, which is easier than writing them in a programming language used to write application. Drools provides structured approach to handle complex ever changing area like fraud detection in financial services to personalised content recommendations in e-commerce in degital landscape.</p><h4>Dedicated module that can evolve on its own pace</h4><p>It enables data and logic separation, which means you can change the rules without changing the application. It allows domain experts with directly participate. Fits well into micro service pattern (develop and deploy independently)</p><h4>Compliance and Auditing</h4><p>It provides a centralised management system for business rules, which makes it possible to introduce changes fast and with lower operational cost. Rule engines maintain an audit trail of rule executions, making it easier to track decisions for compliance, debugging, and performance optimisation purposes, Organisations can ensure compliance with industry regulations and internal policies by using rule engines to enforce rules and guidelines throughout their processes.</p><h4>Scale and Performance</h4><p>It is based on the ‘<em>Rete’</em> algorithm (and improved further with <a href="https://access.redhat.com/documentation/en-us/red_hat_decision_manager/7.4/html/decision_engine_in_red_hat_decision_manager/phreak-algorithm-con_decision-engine">phreek</a>), which is a fast and efficient way of matching facts against set of rules defined.Drools is optimised for high-performance. The Rete algorithm is suited to scenarios where forward chaining/inferencing is used to calculate new facts from existing facts, or to filter and discard facts in order to arrive at some conclusion.</p><p>Here are attractive use cases that makes it work well with workflow management solutions.</p><p>Dynamic Workflow Routing<br>Workflow engines can use rule engines to dynamically route activity based on business rules. For instance, in a claims processing workflow, a rule engine can decide whether a claim should be automatically approved, reviewed by an expert, or flagged for investigation. Many a times not possible do at design time while designing process definition.</p><p>Decision Points in Workflow:<br>Workflow engines can integrate rule engines at decision points within a workflow to make data-driven choices. For instance, in a loan approval process, a rule engine can assess an applicant’s creditworthiness and determine whether to approve or deny the loan and this can have lot external factors along with contextual information making it difficult to decide at design time.</p><h3>Why not Rule Engine?</h3><p>Simple and Static Rules : if-else is good enough for something that changes rarely. It will be overkill to introduce OR use Drools. (Its all Java as well)</p><p>Real-time Constraints — Not for sub second systems.</p><p>Learning Curve — Its not free there is domain to learn and commit which can be steep as well depending on the usage.</p><p>ETL use cases — It should be used where state has to be maintained OR large scale processing required.</p><h3>Glimpses from Drools</h3><p>Here few samples are provided to get the feel of Drools DSL. ‘when’ is predicate and ‘then’ is for actions. The main purpose of rule actions is to to insert, delete, or modify data in the working memory of the decision engine. Effective rule actions are small, declarative, and readable. The best would to take these results back to REST API and make decisions based on the result rather than doing complex operations. (Examples were picked from JBoss Drools Documentation)</p><pre>rule &quot;Underage Loan Rejection Rule&quot;<br> // Example showing rule working on LoanApplication and Application objects<br>  when<br>    application : LoanApplication()<br>    Applicant( age &lt; 21 )<br>  then<br>    application.setApproved( false );<br>    application.setExplanation( &quot;Underage&quot; );<br>end<br><br>rule &quot;Give free parking and 10% discount to over 60 Golden customer and 5% to Silver ones&quot;<br>  /**<br>   * A complex example involving more predicates and code jumps.<br>   */<br>  when<br>    $customer : Customer( age &gt; 60 )<br>    if ( type == &quot;Golden&quot; ) do[giveDiscount10]<br>    else if ( type == &quot;Silver&quot; ) break[giveDiscount5]<br>    $car : Car( owner == $customer )<br>  then<br>    modify($car) { setFreeParking( true ) };<br>  then[giveDiscount10]<br>    modify($customer) { setDiscount( 0.1 ) };<br>  then[giveDiscount5]<br>    modify($customer) { setDiscount( 0.05 ) };<br>end<br></pre><p>Being statically typed, the compilation error provides what actually failed, enabling tested code gets into production. Pattern matching in Drools is like selecting in SQL and using multiple objects is like joining in SQL.</p><p>Spreadsheet Integration</p><p>Drools has native integration with spreadsheets (XLS, XLSX), Rules can be defined in tabular format. It follows specifc pattern for for specifying rules. The excel gets converted into Drools file (.drl). Here is a simple example. (copied from blog of JVM’s man of all seasons). Its scalable in the way that it can be driven through simple UI, Excel and working with learning Drools <a href="https://docs.jboss.org/drools/release/latestFinal/drools-docs/drools/language-reference/index.html">Rule DSL</a> which is quite complex.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PdxH5DedZtxch8oU77u8bw.png" /></figure><p>Template Integration</p><p>Dynamic rules can be generated using templates, Thymeleaf, FreeMarker are a good fit. The native template engine is deprecrated by Drools.</p><p>Complex event processing</p><p>Drools also supports CEP, a real-time extracting information from events as they arive allowing to respond as quickly as possible. One need to be careful as it can get quite complex, probably full fledged stream processing engines like Kafka-Streams, Flink or Spark which are designed ground up for handling the scale.</p><pre>rule &quot;Sound the alarm if temperature rises above threshold&quot;<br>when<br>  TemperatureThreshold($max : max)<br>  Number(doubleValue &gt; $max) from accumulate(<br>    SensorReading($temp : temperature) over window:length(100),<br>    average($temp))<br>then<br>  // Sound the alarm.<br>end<br><br>rule &quot;Weekdays are high priority&quot;<br>  calendars &quot;weekday&quot;<br>  timer ( int:0 1h )<br>  when<br>    Alarm()<br>  then<br>    send( &quot;priority high - we have an alarm&quot; );<br>end<br><br>rule &quot;Weekends are low priority&quot;<br>  calendars &quot;weekend&quot;<br>  timer ( int:0 4h )<br>  when<br>    Alarm()<br>  then<br>    send( &quot;priority low - we have an alarm&quot; );<br>end</pre><h3>Sample use cases</h3><p>Policy Decisions:</p><p>Employees can request a maximum of “10” vacation days per year.<br>Sick leave can only be taken if a “doctor’s note” is provided.<br>Grant access to confidential documents only to employees with a “specific security clearance”.<br>Automatically lock a user’s account after “three” failed login attempts.</p><p>E-Commerce Recommendations</p><p>If a customer frequently buys books, “recommend related book titles”. Update Inventory.<br>Cross-sell complementary products, such as “batteries” for electronic gadgets, when a customer adds a specific item to their cart.<br>If the customer chooses expedited shipping, ensure delivery within 2 business days.<br>Offer free shipping for orders over “5000₹”.<br>Send a reminder email to customers 15 days before their subscription expires.<br>If a customer cancels their subscription, offer a 20% discount to retain them.<br>Build user segments based on buying and browsing patterns that can be used for targeted ads. Ex: “iPhone Buyers”, “High worth Individuals” etc…<br>Define rules defining traits GoldenCustomer (having balance &gt; ₹1,00,000)</p><h4>Industry Use cases</h4><p>Credit Decisioning. <br>Banks and financial institutions can use rule engines to automate credit decision processes. Rules can be defined to evaluate creditworthiness based on factors like income, credit score, and debt-to-income ratio.</p><p>Insurance Underwriting.<br>In the insurance industry, rule engines can determine policy eligibility and pricing by considering factors such as age, health history, and coverage requirements.</p><p>Fraud Detection<br>Rule engines are valuable for real-time fraud detection. They can analyze transactions and flag suspicious activities based on predefined rules, helping to prevent fraudulent transactions.</p><p>Healthcare Diagnosis.<br>In healthcare, rule engines can assist in diagnosing diseases and recommending treatment plans. Rules can be based on patient symptoms, medical history, and test results.</p><p>Inventory Management.<br>Retailers can optimize inventory levels by using rule engines to set rules for restocking products based on factors like historical sales data, seasonality, and supply chain constraints.</p><p>Dynamic Pricing.<br>E-commerce platforms can implement dynamic pricing strategies using rule engines. Rules can adjust prices based on factors like demand, competitor pricing, and customer behavior.</p><p>Customer Support Routing.<br>Rule engines can route customer support requests to the most appropriate agent or department based on customer characteristics, issue severity, and agent availability.</p><p>Personalised Content Recommendations.<br>Media and content platforms can provide personalised recommendations to users based on their viewing history, preferences, and behavior.</p><p>Supply Chain Optimisation<br>Rule engines can optimise supply chain logistics by dynamically routing shipments, optimising warehouse operations, and managing inventory levels based on demand fluctuations.</p><p>Tax and Legal Compliance<br>Accounting and legal firms can use rule engines to ensure compliance with changing tax laws and regulations, reducing the risk of errors and penalties.</p><h3>References</h3><p><a href="https://www.drools.org/learn/dmn.html">Drools DMN engine, an open source Java™ providing full runtime support for Decision Model and Notation (DMN™) models at Conformance level 3</a></p><p><a href="https://www.baeldung.com/drools">Introduction to Drools | Baeldung</a></p><p><a href="https://docs.jboss.org/drools/release/latestFinal/drools-docs/drools/language-reference/index.html">https://docs.jboss.org/drools/release/latestFinal/drools-docs/drools/language-reference/index.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b6b7b4737f38" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bloom Filters, use cases and implementation]]></title>
            <link>https://pmanvi.medium.com/bloom-filters-use-cases-and-implementation-33ef3803c416?source=rss-9398dbbcf83a------2</link>
            <guid isPermaLink="false">https://medium.com/p/33ef3803c416</guid>
            <dc:creator><![CDATA[Praveen Manvi]]></dc:creator>
            <pubDate>Sat, 01 Apr 2023 19:24:14 GMT</pubDate>
            <atom:updated>2023-04-01T19:24:14.913Z</atom:updated>
            <content:encoded><![CDATA[<p>Use cases and Redis based implementation of Bloom Filter service</p><h4><strong>What is bloom filter?</strong></h4><p>A Bloom filter is a probabilistic data structure that is used to quickly and efficiently test whether an element is a member of a set. It is named after Burton Howard Bloom(1970). It’s intended to save space and better performance.</p><p>Because a Bloom filter is probabilistic, the <strong>possibility of false positives</strong> is inbuilt. i.e. there may be elements that are not in the set that are incorrectly identified as members of the set. However, <strong>there are no false negatives</strong>. That is, if an element is in the set, it will always be identified as such. If you have use case where no one is going to die under water if the data does not exist. The use cases where quickly and efficiently test whether an element is a member of a set, but where it is not necessary to have perfect accuracy. We will discuss about these cases and implementations where fast lookups can make are break applications.</p><p>The idea of a ‘Bloom filter’ is to use a fixed-size bit array and a set of hash functions. When an element is inserted, the hash functions are applied to the element to generate a set of indices in the bit array. Each index is then set to 1. To test whether an element is a member of the set, the hash functions are again applied to the element to generate a set of indices in the bit array. If all of the bits at those indices are set to 1, then the element is probably a member of the set. If any of the bits are not set to 1, then the element is definitely not a member of the set.</p><p><strong>What are Use cases?</strong></p><p>Although the main motivation is faster lookups with minimal storage requirements ,we need to understand about the limitation and cost of it. <strong>We can definitely say YES but not confidently NO</strong>. Its like JPEG may not give pixel perfect stuff but it does give the picture.</p><blockquote>Maintaining Invalid set of stateless tokens/api-keys that are time sensitive</blockquote><p>In server side application, JWT offers best in class stateless, scalable option for building cloud native applications for passing on permissions and security server need not store its completely passthrough. One of the downside there is no way to tie with user’s logout (or any other actions like removing users etc…), the token will be valid by timeout specified in JWT, Usually this limitation forces us to keep timeout very low to minimise the impact and also deal with refresh tokens. We can overcome this by using maintaining all the logged-out tokens and user de-activated/removed tokens in a bloom filter providing us ability to deal with such values and keep the timeout matching with session timeout from UI OR as requested by API user. It’s cheap to store and we can reject these requests.</p><blockquote><em>De-duplication of Ids for Kafka streams and highly index sensitive stores like elastic search where inserts, mutating/updating incurs high cost.</em></blockquote><p>In kafka and storing data in elastic search, most of the time data comes in the form of CSV, flat files, data base changes through Change Detection Capture (CDC) techniques which are prone to duplicate data ingestion. Generally the cost is high mutable operations in index sensitive data store. Creating hash and storing them in bloom filter can save all those operations.</p><blockquote>Managing cohorts</blockquote><p>In AdTech the most important aspect successful ad campaign is building effective cohorts (a group of people with a shared characteristic) and targeting them efficiently. Time, Gender, GeoLocation, Age etc… aren’t sufficient anymore. For that we need to have high write throughput and very quick lookup for incoming request belongs to which cohort. There can be lot many use cases like routing of api servers to premium and non premium servers etc.</p><h4>Design and Implementation</h4><p>The intention here to design and implement the this feature more easily accessible to large number of developers to make use directly. Hence this is developed with spring boot (most popular) and Kotlin (Best JVM language). Other than the use cases that I described above developers looking for spell checking, network packet filtering, and DNA sequence analysis should also find this useful</p><p>Redis is a popular in-memory distributed data store that is frequently used as a cache or for fast access to frequently used data. One of the features that Redis provides is the ability to create and manage bloom filters. Redis provides built-in support for bloom filters. Redis has a command called BF.ADD, which adds an element to a bloom filter, and a command called BF.EXISTS, which tests whether an element is a member of a bloom filter. These commands are very fast and efficient, and they can be used to implement a bloom filter with very little code. Redis provides many features that make it easy to manage and monitor bloom filters and BF.DEBUG can be used to diagnose issues with bloom filters.</p><p>Lua script helps us to execute the operations in server side in Redis akin to stored procedure, when we are checking 100s of cohorts where we can get list with single remote command.</p><p>For simple single JVM application hat don’t require high scalability and performance, Google’s Guava’s bloom filter implementation is designed to be easy to use and customisable. It provides a simple API for adding elements to a bloom filter and checking if an element is likely to be a member of the filter. Guava’s bloom filter also supports configurable parameters such as the expected number of elements and the desired false positive rate.</p><p>A separate service is written offering both ingestion and search capabilities of the use-cases that has been listed above. SpringBoot/Kotlin/<strong>Redis</strong> and UI with Thymeleaf/Htmx stack will include both the implementations.</p><p>&lt;TODO&gt;</p><h4><strong>References</strong></h4><p><a href="https://www.youtube.com/watch?v=Z9_wrhdbSC4https://redis.com/redis-best-practices/bloom-filter-pattern/https://medium.com/lydtech-consulting/kafka-deduplication-patterns-1-of-2-ef0371a3331b">https://www.youtube.com/watch?v=Z9_wrhdbSC4</a></p><p><a href="https://www.youtube.com/watch?v=Z9_wrhdbSC4https://redis.com/redis-best-practices/bloom-filter-pattern/https://medium.com/lydtech-consulting/kafka-deduplication-patterns-1-of-2-ef0371a3331b">https://redis.com/redis-best-practices/bloom-filter-pattern/</a></p><p><a href="https://www.youtube.com/watch?v=Z9_wrhdbSC4https://redis.com/redis-best-practices/bloom-filter-pattern/https://medium.com/lydtech-consulting/kafka-deduplication-patterns-1-of-2-ef0371a3331b">https://medium.com/lydtech-consulting/kafka-deduplication-patterns-1-of-2-ef0371a3331b</a></p><p><a href="https://redis.com/blog/bloom-filter/">https://redis.com/blog/bloom-filter/</a></p><p><a href="https://llimllib.github.io/bloomfilter-tutorial/">Bloom Filters by Example</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=33ef3803c416" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pitch for Kotlin]]></title>
            <link>https://pmanvi.medium.com/pitch-for-kotlin-6bb0d7002995?source=rss-9398dbbcf83a------2</link>
            <guid isPermaLink="false">https://medium.com/p/6bb0d7002995</guid>
            <category><![CDATA[kotlin]]></category>
            <dc:creator><![CDATA[Praveen Manvi]]></dc:creator>
            <pubDate>Sun, 02 Jan 2022 14:05:48 GMT</pubDate>
            <atom:updated>2024-09-09T17:55:09.986Z</atom:updated>
            <content:encoded><![CDATA[<h4>Pitch for Kotlin — 2022</h4><blockquote>Kotlin outperforms Java in most of the aspects.</blockquote><p>Kotlin is a multi-paradigm (object-oriented and functional) language that aims to enable solve software problems efficiently. It is statically typed, meaning, compilation getting many problems without waiting for runtime. Kotlin is a formidable language for JVM developers with highest ROI (with multiplier effect of the code reuse in Web, Mobile, Server, Desktop and Native) compared to alternatives. However Java speeding up with language innovation with faster releases and recent enthusiasm with highly performant ‘Go’ and ‘Rust’, Kotlin deserves a re-look. The writeup articulates the advantages from different stake holder perspective. This comes from experience of execution of multiple projects in Kotlin and also following advancement in this space closely.</p><h3>What’s for me? Executives</h3><p>In general manager love metrics and wants decisions are data driven. In general ideas are useless unless used, execution is real value, until then its limbo.</p><h4>Reduced Project Timelines</h4><p>Kotlin’s brevity benefits mean the language can solve more problems with fewer lines of code. Kotlin’s concise nature allows developers to read and modify code more efficiently which can reduce a project’s time-to-market. Kotlin products use fewer lines of code compared to Java (30–40% less code). I can assert these numbers from personal experience of building spring boot based micro services. Not only does less code reduce development time, but fewer lines of code also support maintenance needs. There is less surface area for bugs to hide and the enhanced readability of the language makes locating bugs easier for developers, resulting in fewer crashes and system failures.</p><h4>Fewer App Crashes</h4><p>Kotlin’s type system is aimed to eliminate the ambiguity of null reference from the code (known as a billion dollar mistake). Anyone who has worked with Java/C++ aware of NullPointerExceptions, they are not only hard forces developers to be as defensive as possible with if(not null) code, especially dealing with code that they don’t have control. Probably Kotlin is the only language that solves this <a href="https://kotlinlang.org/docs/null-safety.html">brilliantly</a>.</p><h4>Last mover advantage</h4><p>Other than few ideas like null management, extension methods, lambda treatment… most of the ideas were influenced by other languages which is a good thing and advantage for us as we can rely tried and tested concepts. That’s a great win for development team. Jetbrains folks are brilliant at developing IDEs and probably understands developers and their needs than anyone else with 20+ years of experience in this field. <em>Kotlin developers are in safe hands.</em></p><h4>Results in more type safe code</h4><p>Type safe code has real business benefits with reduced maintenance cost. P.S. It works against the lazy developers who love to passing around JSON blobs, or maps of &lt;String,Any&gt;, Its the running cost that’s going to high and no developers love to debug/maintain this type of code and increase in testing cost. Hence its more work for developers in greater common good in a good way of course. It results in attrition safe code.</p><h3>What’s for me? Python/Go Developers</h3><p>Modern language constructs that are available in Python/Groovy are available in Kotlin.</p><p>Kotlin can get quite close to the compactness of Python’s list/dict/set comprehensions. Since there is huge eco system, we have answers for almost on any topic with single search on web (google, stackoverflow). Most of java solutions are copy paste-able in IDE, Jetbrains working doubly hard along with Google engineers pushing the adoption. Kotlin code feels more pythonic with type safety. The huge difference is not just in line count, but in cognitive overload.</p><p>Lets take this python example from REPL,</p><pre>&gt;&gt;&gt; noprimes = {j for i in range(2, 8) for j in range(i*2, 100, i)}<br>&gt;&gt;&gt; primes = [x for x in range(2, 100) if x not in noprimes]<br>&gt;&gt;&gt; print(primes)<br>[2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97]</pre><p>Kotlin produces the same results with same code semantics and comprehension, as you can notice we need to be explicit being Set (remove duplicates) and use verbs already available with ranges.</p><pre>&gt;&gt;&gt; val nonprimes = (2..7).flatMap { (it*2..99).step(it).toSet() }<br>&gt;&gt;&gt; val primes = (2..99).filterNot { it in nonprimes }<br>&gt;&gt;&gt; println(primes)<br>[2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97]</pre><p>In general working with data structures being type safe it gives Groovy/<a href="https://kotlinfrompython.com/2017/06/06/lists-dictionaries-iterations-more/">Python</a> feeling which is great.</p><h4>Results in more intelligent code</h4><p>Semicolons, forcing type is archaic. Compilers should help the programmers not the other way round. The more you read/write kotlin it feels intelligent and succinct. As it clearly distinguishes between nullable and non-nullable types, makes code robust and avoid thinking “can it be null”. <br>Please note that Kotlin will continue to benefit for years of effort of software engineering from JVM which solidified after many experiments over the years namely — “Garbage Collection, Battle tested Libraries and Virtual Machine”. There is no important java library, framework that does not offer Kotlin support as first class citizen. It co-opts well with reluctant java developers side by side.</p><h3>What’s for me? Java Programmers</h3><p>First of all Kotlin is Of the Java Developers, by the Java Developers, and for the Java Developers. An essential Kotlin design goal is to create a seamless experience for Java Developers, adding few bits of Kotlin into your existing Java project is a valid approach. Kotlin has incorporated most of the <a href="https://www.amazon.com/Effective-Java-Joshua-Bloch/dp/0134685997">effective java</a> book (probably the best best on java) and makes all these best practices embedded in the language.</p><p>Null Safety, Less verbose makes life lot simpler to code. — Tony Hoare, Inventor of QuickSort termed ‘null reference’ as Billion dollar mistake. IMO this itself is good reason for pick Kotlin over java <br>String Templates makes working with String a breeze with literal that can have embedded expressions.<br>Singletons — makes it super easy with declaring an object. No circus of private constructors and thread safety issues.<br>Data Classes — makes lombok, builder pattern un-necessary and ugly<br>Range Expressions — whole range of options producing super easy readable code.<br>Operator Overloading — makes code simpler to read.<br>Implicit Delegation</p><p>Functional programming is more convenient, free-standing functions and constants can exists and no class needs to defined.Function definitions can be nested. There is language support for immutable value types (data class) and algebraic data types (Sealed class).</p><h3>What’s for me? Spring Developers</h3><p>Largest back end java developers are from Spring eco system. There is no reason to left not to make use of newer Spring reactive framework now a days. Better resource utilisation is good use case for any development. Writing non-blocking web application with Spring WebFlux is quite verbose littered with Mono-s and Flux-es, using the extensions for Kotlin and provided by Spring team (some suspend and Flow), it results compact codebase. As you can see example of swagger api, returning when statement and single statement method.</p><pre>@Operation(description = &quot;Returns Welcome Greeting on Home Page&quot;, summary = &quot;Health check&quot;)<br>@ApiResponses(<br>    value = [<br>        ApiResponse(responseCode = &quot;200&quot;, description = &quot;OK&quot;),<br>        ApiResponse(responseCode = &quot;401&quot;, description = &quot;UnAuthorized&quot;)<br>    ]<br>)<br>@GetMapping(&quot;/&quot;)<br>suspend fun home(): String = &quot;Hello, Good ${greet()}!. from Kotlin Service.&quot;<br><br>fun greet(): String {<br>    val hour = LocalDateTime.now().<em>hour<br>    </em>return when {<br>        hour &lt; 12 -&gt; &quot;Morning&quot;<br>        hour &lt; 17 -&gt; {<br>            &quot;Afternoon&quot;<br>        }<br>        else -&gt; &quot;Evening&quot;<br>    }<br>}</pre><p>All the async operations and dealing with async drivers and data conversions becomes lot easier and it definitely feels better. The extension feature makes it easy and libraries like <a href="https://mapstruct.org/">mapstruct</a> redundant.</p><p>For Ex:</p><pre>data class Post(<br>    val id: Long,<br>    val title: String,<br>    val keywords: String? = null,<br>    val content: String? = null,<br>    val postedAt: LocalDateTime = LocalDateTime.now()<br>)<br><br>data class PostDto(<br>    val id:Long,<br>    val title:String,<br>    val postedAt: String,<br>)<br><br>fun Post.toPostLineItem(): PostDto  =<br>    PostDto(id=this.id, title= this.title, postedAt = postedAt.format(DateTimeFormatter.<em>BASIC_ISO_DATE</em>))</pre><pre>// This is what we will do in controller</pre><pre>fun main() {<br>    val post = Post(id=123, title=&quot;Kotlin is Awesome for SpringBoot&quot;, keywords = &quot;java,spring,kotlin&quot;)<br>    <em>println</em>(post)<br>    <em>println</em>(post.<em>toPostLineItem</em>())<br>}<br>/**<br>Post(id=123, title=Kotlin is Awesome for SpringBoot, keywords=java,spring,kotlin, content=null, postedAt=2022-09-19T13:38:57.054678)<br>PostDto(id=123, title=Kotlin is Awesome for SpringBoot, postedAt=20220919)<br>*/</pre><p>Dealing with same old spring MVC model becomes lot simpler, For ex: statements like ‘model.addAttribute(“kibanaUrl”, kibanaUrl);’ becomes a single line as shown below, as we notice not only we type less but the it’s clear on what is being done.</p><pre>   /**<br>     * Provides env specific admin application URLs for quick access<br>     */<br>    @GetMapping(&quot;/admin&quot;)<br>    fun admin(model: Model): String {<br>            model.asMap()<br>                .putAll(setOf(<br>                        &quot;kibanaUrl&quot; to kibanaUrl,<br>                        &quot;consulUrl&quot; to consulUrl,<br>                        &quot;keyCloakUrl&quot; to keyCloakUrl,<br>                        &quot;sbaUrl&quot; to sbaUrl,<br>                        ))<br><br>        return &quot;admin/index&quot;<br>    }</pre><h3>What’s for me? UI Developers</h3><p>JetBrains Compose Multi platform is promising, it’s a declarative framework, so there is no need updating UI logic as backing data changes. This can also be used on desktop with the benefits of hardware acceleration (<a href="https://skia.org/">https://skia.org/</a>) and for web apps, we can now build production-quality dynamic web experiences using Compose for Web’s DOM API, which supports all browser APIs. It allows to share the code between (android, web, server, ios…) multiple application.</p><h3>Few examples giving glimpse of Kotlin’s power</h3><p>Here we are adding a quick-sort method to existing List class using extension feature. As you can observe we are also getting top 2 values from sorted values with slice. Kotlin makes collections so simple that libraries like Guava, Eclipse collections becomes irrelevant. As you start coding if you think of some functionality you just typing methods you will be positively surprised to see them :).</p><p><a href="https://pl.kotl.in/92846lImL">https://pl.kotl.in/92846lImL</a></p><pre>fun &lt;T : Comparable&lt;T&gt;&gt; List&lt;T&gt;.quickSorted(): List&lt;T&gt; =<br>    if (size &lt; 2) this<br>        else {<br>            val pivot = <em>first</em>()<br>            val (smaller, greater) = <em>drop</em>(1).<em>partition </em><strong>{it </strong>&lt;= pivot<strong>}<br>            </strong>smaller.<em>quickSorted</em>() + pivot + greater.<em>quickSorted</em>()<br>        }<br>    <br>fun main() {<br>        <em>println</em>(<em>listOf</em>(1,4,6,3,2).<em>quickSorted</em>().reversed().<em>slice</em>(0..1))<br>        // Prints [6,4]<br>}</pre><h4>Adding power with DSL to make java library fluent.</h4><p>Following libraries should give hint on how fluent with elegant DSL (Domain Specific Language)support is. <a href="https://www.vaadinonkotlin.eu//dsl_explained/">vaadin-ui</a> , <a href="https://kotlinlang.org/docs/typesafe-html-dsl.html">html-dsl</a> , <a href="https://cloud.google.com/blog/products/application-development/use-grpc-with-kotlin">google-kotlin-rpc-support</a></p><p>Here is a simple example on how we can use ‘infix’ to create DSLs.</p><pre>infix fun &lt;T : Number&gt; T.`%`(percentage: T) = ((percentage.toDouble() / toDouble()) * 100).round()<br>infix operator fun String.times(repeat: Int) = repeat(repeat)<br>fun Double.round(decimalPrecision: Int = 2): Double {<br>    return &quot;%.0${decimalPrecision}f&quot;.format(this).toDouble()<br>}</pre><pre>fun main() {<br>    println(&quot;Pranav &quot; * 3) // prints 3 times string.<br>    println(180 `%` 174) // prints 96.67<br>}</pre><p>Sending email from standard javax.mail library can be fluent and async. Pimp my library pattern (popular with scala) is easy to <a href="https://blog.frankel.ch/beautify-third-party-api-kotlin/">build with kotlin</a>.</p><pre>val emailScope = <em>CoroutineScope</em>(Dispatchers.IO + <em>SupervisorJob</em>())<br><br>val mailerService = MailerService(<br>    emailScope,<br>    SMTPConfig(<br>        host = &quot;smtp.gmail.com&quot;,<br>        port = 465,<br>        smtpSecure = SMTPSecure.<em>SSL</em>,<br>        smtpAuth = true,<br>        username = &quot;auto.emails@examstore.com&quot;,<br>        password = &quot;bla bla &amp; bla&quot;,<br>    )<br>)<br><br>mailerService.compose().<em>apply </em><strong>{<br>    </strong>fromEmail = &quot;auto.emails@examstore.com&quot;<br>    fromName = &quot;From ExamStore Service&quot;<br>    subject = &quot;TEST EMAIL&quot;<br>    to = <em>mutableMapOf</em>(<br>        &quot;praveen.manvi@examstore.com.com&quot; <em>to </em>&quot;Praveen&quot;,<br>        &quot;suresh.puvvala@examstore.com&quot; <em>to </em>&quot;Suresh&quot;,<br>        &quot;praguna.manvi@examstore.com&quot; <em>to </em>&quot;Praguna&quot;,<br>    )<br>    html = &quot;&lt;h1&gt;Hello,&lt;br&gt; message from kotlin emailer service &lt;/h1&gt;&quot;<br><strong>}<br></strong><em>.sendAsync()</em></pre><h4>Working with Concurrency</h4><p>Kotlin coroutines makes writing parallel code very easy task. Although java evolved with robust library <a href="https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html">CompletebleFuture</a>, kotlin makes it breeze and structured concurrency (managing hierarchy, inter thread communication, return values, timeouts, cancelations) super easy for developers.</p><p>To demonstrate parallelism in Java, Venkat Subramaniam (One of my favourite speaker) shows how to use Java’s ExecutorService and Future APIs to calculate the average price for each minute in parallel. This makes the program run faster.</p><p>Now, let’s see how we can achieve the same thing using Kotlin coroutines. Kotlin coroutines are a lightweight, asynchronous programming framework that simplifies concurrent programming. As you can see dealing with Future objects is much easier (async) and parallel code almost looks same as the its serial version &amp; latency is improved by &gt;Nx.</p><pre>import kotlinx.coroutines.*<br>import kotlin.system.measureTimeMillis<br><br>fun main() = runBlocking {<br>    val service = StockService()<br>    val stockPrices = service.getStockPrices()<br>    val timeTaken = measureTimeMillis {<br>        val results  = stockPrices<br>            .map { async { service.getAveragePriceForMinute(it) } }<br>            .awaitAll()<br>        println(&quot;Results: $results&quot;)<br>    }<br>    println(&quot;Coroutine : ${timeTaken}ms&quot;)<br><br>    val timeTakenSerially = measureTimeMillis {<br>        val serialResults = stockPrices.map {<br>            service.getAveragePriceForMinute(it)<br>        }.toList()<br>        println(&quot;Results: $serialResults&quot;)<br>    }<br>    println(&quot;Serially: ${timeTakenSerially}ms&quot;)<br>    /**<br>    Results: [105, 110, 115, 120, 125, 130, 135, 140, 145, 150]<br>    Coroutine : 1017ms<br>    Results: [105, 110, 115, 120, 125, 130, 135, 140, 145, 150]<br>    Serially: 10044ms<br>   */<br>}<br><br>class StockService {<br>    fun getStockPrices(): List&lt;Int&gt; {<br>        return listOf(100, 105, 110, 115, 120, 125, 130, 135, 140, 145)<br>    }<br><br>    suspend fun getAveragePriceForMinute(price: Int): Int {<br>        delay(1000) // 1 sec to simulate calculation<br>        return price + 5<br>    }<br>}</pre><h3>Summary</h3><p>Kotlin is great choice for 2022 and beyond. It has power, community and suitable for server side , mobile , web, data engineering and writing scripts. The availability of vast number of java libraries and developers makes it awesome and sustainable choice for any kind of organisation.</p><p>Kotlin delivers well in ‘less with more’ promise. — ‘Succinctness is power’ — 40% less compared to java in most of the cases. Kotlin’s concise syntax(compared to java), which makes it easier to write and read code saving developers time and it also makes difficult to write code that results in null pointer exceptions.<br>Kotlin is multi paradigm — imperative and functional. <br>Kotlin coroutines helps to create high-performance asynchronous code easier compared to other languages.<br>Kotlin allows to intermix with Java code — Its not ‘take it OR leave it’ with java unlike scala<br>Kotlin allows to create fluent APIs — DSLs <br>Kotlin has first class support from popular frameworks (Spring, Quarkus)</p><p><strong>Will java ever be able to catch up with Kotlin and how about performance of Kotlin compared to Java?</strong></p><p>I don’t think so in terms of language features, language &amp; libraries backward compatibility baggage and inability to remove old style of coding will hamper java for ever. This is why even java adds all features available in Kotlin, it will be pulled down by designs like Vector, Properties (Hashtable), Nullability … Sure, Java is moving fast and please note that all its JVM improvements will be available for Kotlin but its language feature all depends on adoptability. java.util.(Date, Stack, Vector) remain as examples bad design. As seen from multiple benchmarks Kotlin offers pretty much the same runtime performance as Java. In many cases because of programming practices (inlining, better structured concurrency with coroutines), it can perform better compared to java as well. In general there won’t become a problem in production that’s unique to kotlin but not available in java.</p><p><strong>Is it not difficult to find Kotlin developers compared to java?</strong></p><p>True, For maintaining/enhancing large application, java is safe bet anytime. However most of the java developers will become productive (if not proficient) in kotlin in no time. Having gone through scala/groovy hype in the past I can safely claim that kotlin is different and possibly safe and best bet with best in class inter-operability with java and java developers. For green field project running on JVM there is no reason not to use Kotlin. There is learning curve and its worth it. (there is no free lunch anyway).</p><p><strong>What the best minds using doing these days?</strong></p><p>There is a clear trend of companies having best in class engineers are switching from Java to Kotlin. Examples include but not limited to Uber, Pinterest, Evernote, Netflix, Amazon, Trello, Doordash, Square, Coursera and of course Google;</p><p>Some disappointments…</p><p>While defining maps, keys and values are paired with the ‘to’ operator, why not ‘:’, I loved groovy, python for this. Kotlin also changed the extends keyword into the : operator. Reversed type declaration i.e. age:Int instead int age was annoying in the beginning, now it’s fine. Missing of package-private visibility modifier felt confusing. Java 17 is already caught up with latest language/JVM improvements. Java 19 getting virtual threads may reduce importance of co-routines, with these most of the touted advantages of kotlin are not relevant. Please note that there are few missing features (compared to java) like “Ternary Operator”, “Static variable”, “Bitwise operators”. Please note there are alternatives to all and there is rationale why they were not included. Data classes (Records), Sealed Classes, Pattern Matching with switch &amp; Multiline strings. New <a href="https://github.com/openjdk/amber-docs/blob/master/site/design-notes/records-and-sealed-classes.md">Java’s pattern</a> matching arguably more powerful than Kotlin. Some concerns include what happens if Jetbrains pulls the plug and make it closed? Can they make it more aligned to IDE to sell IntelliJ licenses.</p><h4>References</h4><p><a href="https://2024.springio.net/slides/spring-boot-and-kotlin-pain-or-gain-springio24.pdf">https://2024.springio.net/slides/spring-boot-and-kotlin-pain-or-gain-springio24.pdf</a> new addition and probably the best pitch talk on Kotlin.</p><p><a href="https://kotlinlang.org/lp/server-side/case-studies/">Server side adoption</a> , <a href="https://drive.google.com/file/d/1gJnxuLtmHKtLy3QnJzZVzjXSJSk3wk37/view">why kotlin</a><br><a href="https://www.thoughtworks.com/de-de/radar/languages-and-frameworks?blipid=202110072"> Defacto standard for mobile</a><br><a href="https://github.com/Kotlin/kotlin-jupyter">Kotlin for datascience</a> <a href="https://blog.khanacademy.org/kotlin-on-the-server-at-khan-academy/">Khan Academy experiments</a><br><a href="https://kotlinlang.org/education/why-teach-kotlin.html">https://kotlinlang.org/education/why-teach-kotlin.html</a></p><p><a href="https://doordash.engineering/2021/05/04/migrating-from-python-to-kotlin-for-our-backend-services/">why doordash picked kotlin</a></p><p><a href="https://aws.amazon.com/blogs/opensource/adopting-kotlin-at-prime-video-for-higher-developer-satisfaction-and-less-code/">https://aws.amazon.com/blogs/opensource/adopting-kotlin-at-prime-video-for-higher-developer-satisfaction-and-less-code/</a> — amazon prime video</p><p><a href="https://phauer.com/2017/idiomatic-kotlin-best-practices/">idiomatic kotlin</a> and <a href="http://Writing Kotlin Idiomatic Code | by Hootsuite Engineering | Hootsuite Engineering | Medium">idiomatic kotlin</a> should help newcomers to join a religion (kotlin here :) ) following standard rituals.</p><p><a href="https://gi-no.github.io/kotlin-is-like-typescript/">Quick kotlin look up for typescript developers</a></p><p><a href="https://khan.github.io/kotlin-for-python-developers/">Quick kotlin look up for python developers</a></p><p><a href="https://gist.github.com/khatv911/d6f25cd3026b2e51f4a4299dbea0a6de">p.md (github.com)</a> -for java script users</p><p><a href="https://github.com/MindorksOpenSource/from-java-to-kotlin">java to kotlin</a> — nice to pick kotlin fast for java developers.</p><p><a href="https://ttu.github.io/kotlin-is-like-csharp/">https://ttu.github.io/kotlin-is-like-csharp/</a> — Kotlin C# programmers</p><p><a href="https://kotlin.link/">Kotlin is Awesome!</a> — Updated to link to look for awesome stuff with kotlin</p><p><a href="https://kotlinlang.org/docs/coding-conventions.html#colon">Coding conventions | Kotlin (kotlinlang.org)</a> &amp; <a href="https://developer.android.com/kotlin/style-guide">Kotlin style guide | Android Developers</a> &amp; <a href="http://raywenderlich/kotlin-style-guide:%20The%20raywenderlich.com%20Kotlin%20style%20guide%20(github.com)">nice summary</a></p><p><a href="https://tibtof.github.io/kotlin-to-rule-dsls/assets/player/KeynoteDHTMLPlayer.html#117">https://tibtof.github.io/kotlin-to-rule-dsls/assets/player/KeynoteDHTMLPlayer.html#117</a> Kotlin is awesome for building rules last slide covers this power in succinct manner.</p><p>Lead language designer (now former) andrey breslav <a href="https://www.youtube.com/watch?v=Ljr66Bg--1M">brilliant session</a> “shoulders of Giant” what features Kotlin learnt and un-learnt from other languages</p><p>Some quotes Kotlin Heroes.</p><h3>Ty Smith on Twitter: &quot;&quot;Kotlin is now the recommended programming language for server-side JVM usage at Google, set to replace Java while still providing access to a large existing Java ecosystem.&quot;https://t.co/KpIqEwYzpR / Twitter&quot;</h3><p>&quot;Kotlin is now the recommended programming language for server-side JVM usage at Google, set to replace Java while still providing access to a large existing Java ecosystem.&quot;https://t.co/KpIqEwYzpR</p><p>“Languages are often selected by passion, not reason… I’m trying to make Kotlin a language that is loved for a reason.” — Andrey Breslav, Kotlin Lead</p><p>“Kotlin is a great language for server-side development because it is easy to read and write, has a concise syntax, and is compatible with Java libraries and frameworks.” — Venkat Subramaniam, Founder of Agile Developer, Inc.<br>“Kotlin is a game-changer for server-side development. Its clean syntax, null-safety features, and powerful functional programming capabilities make it a pleasure to work with.” — Christina Lee, Senior Software Engineer at Google<br>“Kotlin is an excellent choice for server-side development because it is a modern, concise, and type-safe language that reduces the likelihood of bugs and errors.” — Ray Tsang, Developer Advocate at Google<br>“Kotlin has everything you need for server-side development, including great tooling, excellent library support, and a growing community of developers.” — Hadi Hariri, VP of Developer Advocacy at JetBrains</p><p>End of the day, language should reduce the possibility of to make wrong choices. It’s not about lines of code. JavaScript, Python2 results in lesser code but TypeScript and Python 3 with type safe code, although results in more code but it will be robust, readable and maintainable and hence are preferred. Kotlin is the best choice by reducing wrong code decisions to creep in. Secret sauce of scalable software is automation, avoiding wrong inputs and maintainable. Kotlin fits the bill well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*jZcMFBSoqArEWYbj" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6bb0d7002995" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>