<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kaan Berk ÖZBEK on Medium]]></title>
        <description><![CDATA[Stories by Kaan Berk ÖZBEK on Medium]]></description>
        <link>https://medium.com/@kaanberkozbek?source=rss-2d45040d93b0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*-dKPMDTg6m56x5KyfPVDbg.jpeg</url>
            <title>Stories by Kaan Berk ÖZBEK on Medium</title>
            <link>https://medium.com/@kaanberkozbek?source=rss-2d45040d93b0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 09:44:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kaanberkozbek/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Resilience Notes — AI Production Readiness]]></title>
            <link>https://medium.com/@kaanberkozbek/resilience-notes-ai-production-readiness-a159beb4ced1?source=rss-2d45040d93b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/a159beb4ced1</guid>
            <category><![CDATA[llmops]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[resilience]]></category>
            <dc:creator><![CDATA[Kaan Berk ÖZBEK]]></dc:creator>
            <pubDate>Thu, 20 Aug 2026 14:10:04 GMT</pubDate>
            <atom:updated>2026-08-20T15:01:17.809Z</atom:updated>
            <content:encoded><![CDATA[<h3>Resilience Notes #1 — AI Production Readiness</h3><p>An AI system is not production ready just because it can generates replies. Also, it has to be aware of recognizing when it cannot do so, and gracefully falling back to a safe state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*82RwpYdSMCMU-PiB9VT6Bg@2x.jpeg" /></figure><p>Consider the dashboard for an AI service:</p><p>• Availability: 99.99% — P95 latency: 850 ms — HTTP 5xx: 0.01%</p><p>• Container, GPU, and dependency health: green</p><p>Technically, everything looks normal. At the same time, the system may be:</p><p>• Answering questions about a new product with outdated documents.</p><p>• Producing a convincing explanation instead of completing the requested task.</p><p>• Calling the correct tool with the wrong parameters.</p><p>• Inventing an answer when it should hand the conversation to a human.</p><p>• Performing well in one language while silently degrading in another.</p><p>• Marking conversations as resolved only for customers to reopen them later.</p><p>The model endpoint is up. The product is not reliable.</p><p>For a traditional web service, 200 OK is at least a meaningful intermediate signal: the application processed the request. In an AI system, a fluent, well-formatted, completely incorrect answer also returns 200 OK. Token generation and successful task completion do not have a deterministic relationship.</p><blockquote>The question is: Did it complete right task, in the right context at an acceptable level of risk?</blockquote><h3>The production AI system is larger than the model</h3><p>A typical AI request travels through a chain like this:</p><pre>User or channel<br>    -&gt; policy, identity, and routing<br>    -&gt; prompt and conversation context<br>    -&gt; retrieval, embedding, corpus, and index<br>    -&gt; model provider and model version<br>    -&gt; agent planning and tool selection<br>    -&gt; downstream APIs and data stores<br>    -&gt; guardrails and output processing<br>    -&gt; response, action, refusal, or human handoff<br>    -&gt; user and business outcome</pre><p>Every element can change independently. Code may remain unchanged while a system prompt is edited, a knowledge source expires, an embedding model is upgraded, a provider changes a model behind an alias, a tool schema adds a mandatory field, or traffic shifts toward a new customer segment.</p><p>That creates two classes of failure:</p><p>• <strong>Technical failure:</strong> The system shows a timeout, exception, unavailable dependency, or saturation; the user receives no answer or a slow answer.</p><p>• <strong>Functional AI failure:</strong> The request succeeds and the workflow completes; the user receives a wrong, unsafe, stale, irrelevant, or incomplete outcome.</p><p>Traditional SRE approach tells us to control remain necessary for the technical failure. However, AI Production Readiness adds controls for the functional failures. These two works together.</p><h3>Readiness is a lifecycle</h3><p>A one-time checklist quickly becomes outdated as the model, prompt, data, tools, or user population evolves. What is needed instead is a <strong>continuous lifecycle</strong>:</p><p><strong>Inventory → Criticality → Design Review → Golden Set → Evaluation<br> → Resilience Testing → Release Gate → Production Monitoring<br> → Incident Learning → Golden Set</strong></p><p>This creates a continuous feedback loop with three important properties.</p><p>First, <strong>quality, safety, and recovery are built in from the start</strong>, with clear requirements defined before the first production release. Second, <strong>the system is continuously validated in production</strong> by sampling and evaluating representative real-world traffic. Third, <strong>production failures drive improvement</strong>: each failure is turned into a new test, control, or chaos scenario, so that lessons learned become part of the engineering process rather than remaining only in a postmortem.</p><p>AI Production Readiness should therefore be <strong>embedded across the existing Service Resilience lifecycle</strong>, rather than treated as a separate certification activity. Readiness is a shared responsibility: application engineering, business owners, data owners, security, model risk or AI governance, observability, and SRE each contribute and maintain the evidence required to operate the service safely and reliably in production.</p><h3>Start with criticality and unacceptable outcomes</h3><p>Before selecting evaluation or observability tools, <strong>define what the system is expected to do and the conditions under which it can operate safely</strong>.</p><p>For each use case, define:</p><ul><li>The task the system is expected to perform.</li><li>The user and business outcome that defines success.</li><li>The actions the system is permitted to take.</li><li>The decisions that require human approval.</li><li>The outcomes that are unacceptable under any circumstance.</li><li>The maximum tolerable impact, duration, and data exposure.</li><li>The safest operating mode when confidence is low or a dependency fails.</li></ul><p>Risk should be assessed at the <strong>use-case level, not simply at the model level</strong>. The same model may summarize an internal document, recommend a support article, change a customer address, approve a payment, or control production traffic. These activities have fundamentally different consequences and therefore should not be governed by the same release criteria or level of autonomy.</p><p>A practical assessment should consider factors such as <strong>financial and customer impact, regulatory and privacy exposure, reversibility, blast radius, degree of automation, availability of human oversight, and the ability to detect incorrect outcomes</strong>.</p><p>Most importantly, criticality must determine the controls applied to the system. As criticality increases, the requirements should become progressively stronger: broader golden-set coverage, stricter release blockers, reduced autonomy, greater human validation, verified fallback mechanisms, tested kill switches, and more frequent evaluation in production.</p><h3>Why should we use “Golden Dataset”?</h3><p>The golden dataset is an executable contract. A golden dataset should be treated as an <strong>executable definition of expected system behavior</strong>, not as the product’s knowledge base or a collection of ideal answers. It is a versioned set of representative, reviewed scenarios used to determine whether the system behaves correctly under conditions that matter to the business.</p><p>For an initial pilot, <strong>50–150 carefully selected scenarios</strong> may provide more value than thousands of loosely generated examples. The objective is not volume, but meaningful coverage of critical business behavior, important boundaries, and known failure modes.</p><p>Each scenario should contain enough information to determine not only whether the system passed or failed, but also:</p><ul><li>Scenario and business purpose</li><li>Input, conversation history, identity, and relevant context</li><li>Risk tier and user segment</li><li>Expected critical facts</li><li>Required or acceptable sources</li><li>Prohibited claims or behaviors</li><li>Expected retrieval behavior</li><li>Expected tool, parameters, and authorization state</li><li>Expected final outcome, refusal, or human handoff</li><li>Deterministic blockers</li><li>Semantic evaluation criteria and thresholds</li><li>Owner, reviewer, dataset version, and last review date</li></ul><p>Golden cases should come from multiple sources because each reveals a different type of weakness:</p><ol><li><strong>Real production interactions</strong>, after appropriate privacy controls and review, capture how users actually express intent, including ambiguity and unexpected phrasing.</li><li><strong>Subject-matter expert scenarios</strong> establish the business, policy, and procedural expectations the system must respect.</li><li><strong>Synthetic variations</strong> broaden coverage across language, locale, format, context, and boundary conditions.</li><li><strong>Adversarial scenarios</strong> test resistance to prompt injection, data exfiltration, excessive agency, unauthorized actions, and policy bypass.</li><li><strong>Incident-derived scenarios</strong> turn production failures into permanent regression tests, reducing the chance that the same failure silently returns.</li></ol><p>The golden dataset should be <strong>versioned independently</strong> from the model, prompt, application, knowledge index, and evaluator. Without independent versioning, it becomes difficult to determine whether a change in evaluation results reflects a product regression or simply a change in the test itself.</p><p>AI systems should not depend on a single evaluator to determine production readiness. Each evaluation method has different strengths and limitations: deterministic tests are reliable but narrow, model-based evaluation can assess meaning and context but introduces variability, and human review provides informed judgment but cannot be applied continuously at scale.</p><p>A robust evaluation system therefore <strong>combines multiple layers of evidence</strong>, from technical correctness to business outcomes. Release decisions should be based on the combined result, with critical failures acting as explicit blockers regardless of the overall score.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VYqIAxW5vrTzSMDe4VqRcw.png" /></figure><h3>Separate retrieval from generation</h3><p>Retrieval and generation should be evaluated as <strong>two distinct parts of the system</strong>. Assessing only the final response makes it difficult to identify where a failure originated. A convincing answer may hide poor retrieval, while an incorrect answer may be attributed to the model even when the required information was never provided to it.</p><p><strong>Retrieval evaluation</strong> should determine whether the system supplied the right evidence:</p><ul><li>Was the authoritative source retrieved?</li><li>Did it appear within the required top-k results?</li><li>Was the source current, approved, and accessible to the user?</li><li>Did chunking preserve the information required to complete the task?</li><li>Did metadata and access filters select the correct tenant, region, product, and policy version?</li></ul><p><strong>Generation evaluation</strong> should determine whether the system used that evidence correctly:</p><ul><li>Is the response supported by the retrieved evidence?</li><li>Are all critical facts represented accurately?</li><li>Are citations correct and attributable to the appropriate sources?</li><li>Are conflicting sources identified and handled appropriately?</li><li>Does the system abstain or escalate when the available evidence is insufficient?</li></ul><p>Keeping these evaluations separate makes failures easier to diagnose and correct. <strong>Retrieval failures</strong> typically point to indexing, ranking, chunking, metadata, freshness, or access-control issues. <strong>Generation failures</strong> point instead to how the available evidence is interpreted and expressed, and may require changes to the prompt, model, evaluation criteria, or post-processing.</p><p>The objective is not simply to determine <strong>whether the system failed, but where and why it failed</strong>.</p><h3>Evaluate the full agent execution path</h3><p>For agentic systems, evaluating only the final response is not enough. An agent may produce an acceptable result after following an unsafe or inefficient path. The final answer can hide excessive tool calls, incorrect parameters, repeated loops, unauthorized actions, or side effects created earlier in the execution.</p><p>Evaluation should therefore cover the <strong>complete execution path</strong>, including:</p><ul><li>Intent identification</li><li>Tool selection</li><li>Parameter accuracy</li><li>Authorization and least-privilege enforcement</li><li>Execution sequence</li><li>Loop and retry limits</li><li>Task completion</li><li>Recovery and outcome verification</li><li>Human handoff and context transfer</li></ul><p>The key principle is to <strong>separate model reasoning from execution authority</strong>. The model can determine what action it believes should be taken, but it should not have unrestricted authority to execute that action.</p><p>The execution layer should apply deterministic controls before an action is performed. These controls determine whether the requested action is permitted, whether the user and system have the required authorization, whether limits apply, whether the action can be reversed, and whether human approval is required.</p><p>This creates an important boundary: <strong>the model proposes; the controlled execution layer authorizes and acts</strong>.</p><h3>Use LLM-based evaluation carefully</h3><p>An LLM can be useful for evaluating qualities that are difficult to capture with deterministic rules, such as relevance, completeness, groundedness, and clarity. However, its judgment should be treated as <strong>one source of evidence, not as an objective measure of truth</strong>. Results can be influenced by the evaluation prompt, rubric design, model family, response style, and changes to the judge model itself.</p><p>A reliable approach combines model-based evaluation with deterministic controls and human judgment:</p><ul><li>Use <strong>deterministic rules</strong> for requirements that must never be violated.</li><li>Use a <strong>separate evaluator model</strong> for criteria that require semantic judgment.</li><li>Define an <strong>explicit rubric</strong> with clear examples of acceptable and unacceptable behavior.</li><li>Use <strong>blind or pairwise comparisons</strong> when relative quality is more reliable than absolute scoring.</li><li>Repeat evaluations when criteria produce unstable results.</li><li><strong>Calibrate against expert human review</strong> on a regular basis.</li><li>Version the <strong>judge model, prompt, rubric, and thresholds</strong> so that evaluation changes are traceable.</li><li>Apply <strong>risk-based human review</strong>, particularly for critical, borderline, or disputed outcomes.</li></ul><p>Disagreement between human reviewers and the evaluator should not simply be treated as evaluation noise. <strong>Persistent disagreement is useful evidence</strong>: it often indicates that the rubric, threshold, or definition of acceptable behavior is unclear and needs to be refined.</p><h3>Turn evaluation into a release gate</h3><p>Evaluation should be part of the <strong>release decision</strong>, not a report reviewed after the fact. Any change that can alter system behavior — including application code, model, prompt, policy, tool definitions, embedding model, knowledge corpus, or retrieval index — should pass explicit readiness criteria before reaching production.</p><p>A practical pipeline can follow this pattern:</p><p><strong>Change introduced</strong><br> → Deterministic tests for schema, grounding, policy, authorization, and resilience<br> → Semantic evaluation for response quality and failure analysis<br> → Adversarial and security testing<br> → Comparison against approved baselines by risk tier and user segment<br> → <strong>READY / BLOCKED</strong> decision with retained evidence<br> → Canary or shadow deployment<br> → Production tracing, feedback, and continuous evaluation<br> → Production failures incorporated into the golden dataset</p><p>Tools such as <strong>Pytest, DeepEval, Promptfoo, and Langfuse or OpenTelemetry-based platforms</strong> can support different parts of this process, but the architecture should not depend on a particular product. Product-specific requirements are best enforced through deterministic tests, while semantic evaluation, adversarial testing, and production analysis can use appropriate open-source or managed solutions.</p><p>More important than the individual tools is a <strong>common evaluation contract</strong>: every layer should use consistent scenario identifiers, artifact versions, risk classifications, and outcome definitions. This makes it possible to trace a release decision back to the exact system configuration and evidence that supported it.</p><h3>What should block a release?</h3><p>A release should be blocked when any non-negotiable production requirement is violated. At minimum, this includes:</p><ul><li>Failure of a critical safety, privacy, authorization, or policy scenario</li><li>Retrieval or exposure of data belonging to an unauthorized user or tenant</li><li>Incorrect or unauthorized parameters for a high-impact action</li><li>Failure to abstain or escalate when human intervention is mandatory</li><li>Regression beyond the permitted threshold for a critical use case or segment</li><li>Missing rollback capability or incompatible artifact versions</li><li>Absence of a verified fallback for a critical dependency</li><li>Insufficient trace context to investigate a production quality or safety incident</li></ul><p>These conditions should be treated as <strong>hard gates rather than weighted metrics</strong>. Aggregate scores are useful for measuring overall quality and detecting trends, but they must not compensate for critical failures.</p><p><strong>A system is READY only when every mandatory control passes.</strong> Ninety-nine successful low-risk cases cannot offset a single unauthorized payment, privacy breach, or other unacceptable outcome.</p><h3>Version and control every AI artifact independently</h3><p>An AI service is made up of multiple artifacts that can change system behavior independently. The model, prompt, policy, retrieval index, embedding model, and tool contracts should therefore each have their <strong>own version identity, deployment history, and rollback path</strong>.</p><p>Tracking only the application version is not sufficient. The same application release may run with different prompts, models, policies, or knowledge indexes and produce materially different outcomes.</p><p>For every production release, retain a complete record of the configuration and evidence used to approve it:</p><ul><li>Application code revision</li><li>Model provider, model name, and resolved model version</li><li>Prompt and policy versions</li><li>Golden dataset and evaluator versions</li><li>Embedding model and retrieval-index versions</li><li>Tool and API schema versions</li><li>Evaluation gate results and required approvals</li><li>Canary or rollout cohort and defined rollback target</li></ul><p>Together, these elements create a <strong>reproducible release record</strong>. When an incident occurs, teams should be able to determine exactly which combination of code, model, prompt, data, policies, and tools was running at that point in time — and restore the last known safe configuration.</p><p>Without this level of traceability, it becomes difficult to determine <strong>what actually changed, why behavior changed, and what should be rolled back</strong>.</p><p>Supporting SLIs should cover six views:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fGoGNmZRiTReFm87EsaVrQ.png" /></figure><h3>Define AI SLIs, SLOs, and error budgets around successful outcomes</h3><p>Traditional availability remains necessary, but it does not show whether an AI service is actually delivering the expected outcome. A system can be available, fast, and technically healthy while still producing incorrect, unsafe, or incomplete results.</p><p>For AI services, the primary SLI should therefore measure <strong>quality-adjusted task success</strong>:</p><p><strong>Quality-Adjusted Task Success</strong></p><p>Quality-Adjusted Task Success = Eligible tasks that reached the correct terminal outcome and passed mandatory quality and safety criteria<br>= All eligible AI tasks</p><p>A task counts as successful only when the <strong>end-to-end outcome is correct</strong>. A technically successful request that produces an incorrect answer, performs the wrong action, violates a mandatory control, or fails to complete the intended task should not contribute to the success rate.</p><p>The definition of an <strong>eligible task</strong> is equally important. Eligibility criteria should be explicit, versioned, and consistent over time. Changes to the denominator should be reviewed with the same discipline as changes to the evaluation criteria.</p><p>Difficult requests must not simply be excluded because they reduce the reported success rate. Otherwise, the metric may improve while the actual user experience deteriorates.</p><p>The objective is to measure <strong>how often the AI service completes the work it is expected to perform, correctly and safely — not simply how often it responds</strong>.</p><h3>Define an AI error budget</h3><p>A conventional error budget measures how much service degradation can be tolerated before reliability objectives are breached. For AI services, the same principle should extend beyond availability and latency to include <strong>quality, safety, and task outcomes</strong>.</p><p>The error budget may be consumed by events such as:</p><ul><li>Unavailable or excessively slow requests</li><li>Incorrect or ungrounded outcomes beyond the accepted threshold</li><li>Missed mandatory escalations</li><li>Unsafe or unauthorized actions</li><li>Failed fallback mechanisms</li><li>Tasks that are subsequently corrected, reopened, or reversed</li></ul><p>Error-budget consumption should not be assessed only at the aggregate service level. It should be possible to analyze burn by <strong>risk tier, task type, customer segment, language, model, prompt, retrieval index, and tool version</strong>. A healthy global average can otherwise conceal persistent failures in a specific high-risk workflow.</p><p>For critical use cases, some events should remain <strong>hard failures rather than simply consuming budget</strong>. A privacy breach or unauthorized high-impact action, for example, should trigger immediate response even when the overall error budget remains healthy.</p><h3>Make the full AI workflow observable</h3><p>During an AI-quality incident, identifying the model is only the beginning. Teams need to reconstruct the <strong>complete path from user intent to final business outcome</strong>.</p><p>A task or conversation trace should connect information such as:</p><p>task_id / conversation_id<br> task_type / risk_tier / user_segment / locale<br> model_provider / requested_model / resolved_model<br> prompt_version / policy_version<br> embedding_model / retrieval_index / document_versions<br> retrieval_scores / source_authority / source_freshness<br> agent / operation / step_count / retry_count<br> tool / schema_version / parameter_hash / authorization_result<br> guardrail_results / evaluation_scores / evaluator_version<br> escalation_reason / fallback_reason / human_override<br> terminal_outcome / delayed_business_outcome / cost</p><p>The objective is to make these signals <strong>correlatable across the entire execution</strong>, rather than storing them as disconnected logs.</p><p>OpenTelemetry-style tracing provides a useful foundation for this approach. A trace can connect model calls with retrieval operations, agent steps, tools, downstream services, evaluation events, and the eventual outcome. Platforms designed for high-cardinality observability, such as Honeycomb, can then support investigation across dimensions that were not known in advance.</p><p>For example, an incident investigator should be able to ask:</p><blockquote><em>Did Turkish enterprise users running prompt version 17 receive outdated policy documents from retrieval index 42 after the embedding-model upgrade?</em></blockquote><p>This level of investigation requires rich context, but <strong>it does not mean recording everything</strong>. Prompts, responses, retrieved documents, and tool parameters can contain personal, confidential, or regulated information. Production telemetry should therefore apply:</p><ul><li>Data minimization</li><li>Masking, hashing, or tokenization where appropriate</li><li>Collector-side filtering before telemetry leaves the controlled environment</li><li>Role-based access controls</li><li>Defined retention periods</li><li>Auditable access and configuration changes</li><li>A documented exception process for preserving evidence during incidents</li></ul><p>The principle is simple: <strong>collect enough context to reconstruct and investigate system behavior, but no more sensitive data than the investigation requires</strong>. An observability system that creates a new privacy or security exposure is itself a production-readiness failure.</p><h3>Continuous evaluation makes drift operationally visible</h3><p>Pre-release evaluation shows how the system behaves against a known set of scenarios. It cannot guarantee that future production traffic, knowledge sources, dependencies, or user behavior will remain the same. Production behavior therefore needs to be <strong>evaluated continuously, not only observed through infrastructure metrics</strong>.</p><p>A continuous evaluation process should:</p><ol><li>Sample representative production interactions according to use-case risk.</li><li>Attach quality and business-outcome labels as they become available.</li><li>Compare current performance with an approved fixed or rolling baseline.</li><li>Analyze results by task, risk tier, customer segment, locale, artifact version, and dependency path.</li><li>Route critical, ambiguous, or low-confidence cases for human review.</li><li>Convert confirmed production failures into curated regression scenarios.</li><li>Update baselines only through a controlled and auditable process.</li></ol><p>Drift should not be reduced to a single metric. Different parts of the system can change independently and should be monitored separately:</p><ul><li><strong>Input drift:</strong> Changes in user intent, language, traffic patterns, and task complexity.</li><li><strong>Context drift:</strong> Changes in retrieval success, source freshness, corpus content, embeddings, or index behavior.</li><li><strong>Output drift:</strong> Changes in correctness, groundedness, citations, structure, policy compliance, or other required response characteristics.</li><li><strong>Trajectory drift:</strong> Changes in tool selection, execution steps, retries, loops, and human handoffs.</li><li><strong>Outcome drift:</strong> Changes in task completion, corrections, reopenings, abandonment, user effort, and downstream business outcomes.</li></ul><p>A detected statistical change should be treated as an <strong>investigation signal, not automatic evidence of degradation</strong>. Changes in traffic may be legitimate — for example, after a product or marketing launch.</p><p>Drift becomes operationally significant when the change is associated with deterioration in <strong>quality, safety, user effort, reliability, or business outcomes</strong>. The objective is therefore not to keep every distribution unchanged, but to detect when a changing environment begins to affect the service outcomes that matter.</p><h3>Test AI resilience with failure injection</h3><p>AI Chaos Engineering should test behavioral degradation as well as infrastructure failure. Run controlled scenarios such as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*79vtXyVoRe_hEnCEzzrq9Q.png" /></figure><p>A resilience test should verify <strong>the complete recovery outcome, not simply that a fallback mechanism responded</strong>. The test should confirm that the user reached a safe terminal state, no unauthorized or prohibited action occurred, system and business state remained consistent, and normal service was restored without introducing additional failures.</p><h3>Define fallback as the safest acceptable operating mode</h3><p>Fallback should not be limited to routing the same request to another model. That approach may improve availability without addressing the underlying failure. If the problem is caused by stale retrieval, incorrect context, a broken tool, or an authorization issue, another model may reproduce the same failure.</p><p>The appropriate fallback should therefore depend on <strong>what failed and what the system can still do safely</strong>. Depending on the use case, this may include:</p><ul><li>Switching model, provider, or region</li><li>Returning a verified cached or deterministic response</li><li>Moving to retrieval-only or read-only operation</li><li>Disabling tool actions while preserving guidance or explanation</li><li>Restricting the service to a narrower supported workflow</li><li>Asking the user for clarification</li><li>Requiring human approval or transferring the task to a human</li><li>Returning a safe refusal</li><li>Disabling the affected capability through a kill switch</li></ul><p>Fallback effectiveness should be measured by the <strong>result it produces, not by whether the fallback was invoked successfully</strong>:</p><p><strong>Fallback Success</strong></p><p>= Fallback invocations that reached a safe and valid terminal outcome<br> ÷ All fallback invocations</p><p>Relevant measures include fallback trigger rate, time to fallback, fallback latency, retry and loop count, completeness of context transfer, task success after fallback, and successful restoration of normal service.</p><p>A fallback should also be tested under realistic failure conditions. <strong>A fallback that has never been exercised is an assumption, not a verified control.</strong></p><h3>Create an AI-specific incident lifecycle</h3><p>AI incidents may not begin with an infrastructure alert. The first indication may instead be a customer complaint, declining evaluation score, unexpected agent behavior, increased human escalation, a cost anomaly, or a gradual reduction in task success.</p><p>The incident taxonomy should therefore cover both traditional operational failures and AI-specific behavior, including:</p><ul><li>Model availability or latency</li><li>Quality regression</li><li>Hallucination or unsupported claims</li><li>Retrieval failures or stale knowledge</li><li>Security, privacy, or prompt-injection events</li><li>Agent or tool failures</li><li>Cost or token anomalies</li><li>Compliance or policy violations</li></ul><p>The response lifecycle should be explicit:</p><p><strong>Detect → Classify → Contain → Recover → Verify → Learn</strong></p><p><strong>Containment</strong> limits further impact. Depending on the failure, this may involve restoring a previous prompt, rolling back a retrieval index, disabling a tool, reducing system autonomy, requiring human approval, restricting responses to verified content, or disabling the affected capability entirely.</p><p><strong>Recovery</strong> restores an acceptable operating state, but a healthy endpoint alone is not sufficient evidence. The failed scenarios should be rerun against the golden dataset and representative production traces. Teams should verify that retrieval, tool execution, fallback behavior, system state, and the final user outcome have all returned to the expected condition.</p><p><strong>Learning</strong> should turn the incident into concrete improvements to the system. This may include:</p><ul><li>A new or updated golden scenario</li><li>A deterministic release blocker where possible</li><li>A revised evaluation rubric or threshold</li><li>A new resilience or chaos scenario</li><li>An updated runbook or ownership model</li><li>A verified implementation of the corrective control</li><li>A defined date for retesting the corrective action</li></ul><p>This makes incident learning <strong>executable rather than documentary</strong>. Closing an action item is not sufficient evidence of improvement. A stronger standard is that <strong>the failed behavior can no longer pass the release gate, and the recovery path has been successfully retested</strong>.</p><h3>Use a readiness scorecard, but never average away a blocker</h3><p>A scorecard helps teams compare services and expose missing evidence. One balanced structure is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iAH9U1hVBOccOCWGOvKRWA.png" /></figure><h3>Make production readiness a set of mandatory conditions</h3><p>A weighted readiness score can help teams identify weaknesses and prioritize improvement, but it should <strong>not determine whether a service is allowed into production</strong>. Critical controls cannot be compensated for by strong performance elsewhere. Excellent observability cannot offset missing authorization controls, just as a high semantic-quality score cannot compensate for an untested recovery path.</p><p>Production readiness should therefore use <strong>AND logic</strong>, with mandatory conditions that must all be satisfied:</p><p><strong>PRODUCTION READY =</strong></p><p>Service SLOs are met<br> <strong>AND</strong> critical deterministic tests pass<br> <strong>AND</strong> risk-based evaluation thresholds are met<br> <strong>AND</strong> all must-pass safety scenarios pass<br> <strong>AND</strong> required observability evidence is available<br> <strong>AND</strong> fallback, rollback, and kill-switch mechanisms are verified<br> <strong>AND</strong> incident ownership and response processes are active</p><p>If any mandatory condition fails, the service is <strong>BLOCKED</strong>, regardless of its aggregate readiness score.</p><p>This creates a clear distinction between <strong>scoring and gating</strong>: scores help determine where improvement is needed; gates determine whether the service is safe and ready to operate.</p><h3>Clarify ownership before the first incident</h3><p>AI reliability spans multiple technical and business domains. Production readiness therefore requires <strong>shared accountability with clearly defined ownership</strong>, established before the service goes live.</p><ul><li><strong>Application or product engineering</strong> owns the end-to-end workflow, integrations, implementation, and release artifacts.</li><li><strong>Business owners and subject-matter experts</strong> define critical facts, expected outcomes, business rules, and unacceptable behavior.</li><li><strong>Data owners</strong> are accountable for source authority, freshness, lineage, quality, and access requirements.</li><li><strong>Security</strong> owns security controls, privacy requirements, identity, authorization, and least-privilege enforcement.</li><li><strong>AI governance or model risk</strong> owns risk classification, required model evidence, evaluation expectations, and policy conformance.</li><li><strong>SRE or Service Resilience</strong> owns service SLOs, dependency resilience, incident mechanisms, recovery procedures, fallback exercises, and recovery evidence.</li><li><strong>Observability owners</strong> define trace and telemetry standards, telemetry governance, and the capabilities required to investigate production behavior.</li></ul><p>These responsibilities should converge at the production gate. <strong>No individual team should certify “AI ready” on behalf of the entire service.</strong> Each accountable owner should provide evidence for the controls within its domain.</p><p>At the same time, shared accountability should not mean ambiguous decision-making. Every AI service should have <strong>one clearly identified service owner with the authority to approve, block, or stop production operation</strong> based on the evidence provided by the accountable teams.</p><h3>Measure whether production readiness is improving</h3><p>A production-readiness program should be measured by <strong>evidence of better system behavior</strong>, not by the number of assessments, controls, or governance documents produced.</p><p>Useful indicators include:</p><ul><li>Percentage of critical use cases covered by a current golden dataset</li><li>Risk-weighted evaluation pass rate</li><li>Regression escape rate from pre-production into production</li><li>Percentage of critical dependencies with a tested fallback</li><li>Fallback success and recovery-verification rate</li><li>Mean time to detect an AI-quality incident</li><li>Mean time to restore an acceptable user outcome</li><li>End-to-end trace completeness</li><li>Human-escalation precision, recall, response time, and context quality</li><li>Repeat-incident rate</li><li>Percentage of corrective actions successfully retested</li><li>Cost per successful task rather than cost per model call</li></ul><p>Together, these measures answer a more important question than whether governance activities have been completed: <strong>is the service becoming more capable of preventing failure, containing its impact, recovering safely, and avoiding recurrence?</strong></p><h3>What the major industry approaches have in common</h3><p>The major industry approaches differ in terminology, tooling, and implementation, but they increasingly point toward the same operating model.</p><p>Platforms such as <strong>Microsoft Foundry, Google Vertex AI, Amazon Bedrock, OpenAI Evals, and Anthropic’s evaluation guidance</strong> treat evaluation as an ongoing engineering capability rather than a one-time benchmark. Evaluation is increasingly connected to datasets, model and agent behavior, release comparison, production traces, and monitoring.</p><p>Frameworks such as <strong>NIST AI RMF, ISO/IEC 42001, and the OWASP Top 10 for LLM Applications</strong> add the governance and risk perspective: understand the use case, establish accountability, identify and document risk, test misuse scenarios, protect information, constrain system authority, and retain evidence.</p><p><strong>Google’s SRE guidance</strong> reinforces an important principle for autonomous operations: probabilistic reasoning should not automatically translate into production authority. Investigation, human-approved action, bounded autonomy, least privilege, rate limits, dry runs, circuit breakers, and emergency-stop mechanisms provide progressively stronger boundaries around what an AI system can do.</p><p><strong>Intercom’s production workflow</strong> demonstrates the learning cycle: observe real interactions, convert failures into reproducible scenarios, test those scenarios before subsequent releases, deploy progressively, and verify the resulting user outcome.</p><p><strong>Honeycomb’s observability approach</strong> adds the ability to connect model, prompt, retrieval, tools, dependencies, and outcomes within the same investigation context, making behavioral changes and drift easier to diagnose.</p><p>The common principle is not that any single vendor or framework has solved AI reliability. It is that <strong>evaluation, release control, production evidence, safe degradation, recovery, and incident learning need to operate as one connected system</strong>.</p><h3>Conclusion</h3><p>AI Production Readiness is not a model benchmark, a red-team report, an SRE checklist, or an observability dashboard. It is the <strong>evidence that an end-to-end AI service can consistently produce its intended outcome — and enter a known, safe state when it cannot</strong>.</p><p>Before production, the team should be able to answer:</p><ul><li>Which tasks can the system perform reliably, and for which users and segments?</li><li>Which outcomes are unacceptable even once?</li><li>Which golden scenarios demonstrate the expected behavior?</li><li>Which failures automatically block a release?</li><li>Can retrieval and generation failures be identified independently?</li><li>Are tool selection, parameters, authorization, execution path, and terminal state evaluated?</li><li>Can every behavioral change be traced to the relevant model, prompt, policy, index, embedding model, tool, or code version?</li><li>Does the primary SLO measure successful task outcomes rather than simply successful responses?</li><li>Can production traces connect changes in system behavior with user and business outcomes?</li><li>What happens when the model, retrieval system, guardrail, tool, evaluator, or human-review process becomes unavailable or unreliable?</li><li>Is fallback a tested safe operating mode rather than simply another endpoint?</li><li>Can the affected capability be rolled back, restricted, or stopped quickly?</li><li>Does every confirmed production failure strengthen the tests and controls applied to the next release?</li></ul><p>The durable principle is:</p><p><strong>AI reliability is not measured by how often the model produces an answer. It is measured by how often the system completes the right task correctly and safely and how predictably it responds when it cannot.</strong></p><p>A model-health dashboard can be entirely green while the service is failing its users. <strong>The final measure of readiness is therefore the user and business outcome, not the model response.</strong></p><h3>Sources and further reading</h3><p>• <a href="https://www.honeycomb.io/blog/ai-model-drift">Honeycomb — AI Model Drift: How to Keep Models Reliable</a></p><p>• <a href="https://www.honeycomb.io/blog/we-shipped-ai-product">Honeycomb — So We Shipped an AI Product. Did It Work?</a></p><p>• <a href="https://docs.honeycomb.io/send-data/use-cases/agents">Honeycomb Docs — Instrumenting AI Agents</a></p><p>• <a href="https://learn.microsoft.com/en-us/azure/foundry/how-to/evaluate-generative-ai-app">Microsoft Foundry — Run evaluations from the Foundry portal</a></p><p>• <a href="https://learn.microsoft.com/en-us/azure/foundry/concepts/observability">Microsoft Foundry — Observability in Generative AI</a></p><p>• <a href="https://cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-overview">Google Cloud — Gen AI evaluation service overview</a></p><p>• <a href="https://cloud.google.com/blog/products/ai-machine-learning/introducing-agent-evaluation-in-vertex-ai-gen-ai-evaluation-service">Google Cloud — Evaluate AI agents with the Gen AI evaluation service</a></p><p>• <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/evaluation.html">AWS — Evaluate Amazon Bedrock resources</a></p><p>• <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/evaluation-judge.html">AWS — LLM as a judge</a></p><p>• <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/evaluation-kb.html">AWS — Evaluate RAG retrieval and generation separately</a></p><p>• <a href="https://developers.openai.com/api/docs/guides/evals">OpenAI — Working with evals</a></p><p>• <a href="https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents">Anthropic — Demystifying evals for AI agents</a></p><p>• <a href="https://www.nist.gov/itl/ai-risk-management-framework">NIST — AI Risk Management Framework</a></p><p>• <a href="https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence">NIST — Generative AI Profile</a></p><p>• <a href="https://www.iso.org/standard/42001.html">ISO — ISO/IEC 42001:2023</a></p><p>• <a href="https://genai.owasp.org/llm-top-10/">OWASP — Top 10 for LLM Applications 2025</a></p><p>• <a href="https://sre.google/resources/practices-and-processes/ai-engineering-reliable-operations/">Google SRE — AI in SRE: Engineering Reliable Operations</a></p><p>• <a href="https://www.intercom.com/blog/announcing-monitors-opening-the-ai-black-box/">Intercom — Announcing Monitors: Opening the AI Black Box</a></p><p>• <a href="https://www.intercom.com/blog/announcing-evals-and-releases/">Intercom — Announcing Evals and Releases</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a159beb4ced1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Canary Deployment’larda eBPF ile Hızlı Problem Tespiti ve Argo ile Otomatik Rollback]]></title>
            <link>https://medium.com/@kaanberkozbek/canary-deploymentlarda-ebpf-ile-h%C4%B1zl%C4%B1-problem-tespiti-ve-argo-ile-otomatik-rollback-04ca9a7cd7ae?source=rss-2d45040d93b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/04ca9a7cd7ae</guid>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[resilience]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Kaan Berk ÖZBEK]]></dc:creator>
            <pubDate>Sun, 28 Jun 2026 20:35:32 GMT</pubDate>
            <atom:updated>2026-07-02T06:53:20.973Z</atom:updated>
            <content:encoded><![CDATA[<p>Bir canary deployment’ın ne zaman geri alınması gerektiğini nasıl anlarsınız? HTTP durum kodlarına bakarsınız, hata oranı eşiği aşarsa rollback tetiklersiniz. Peki ya HTTP katmanına bile ulaşamayan paketler? Kernel’ın düşürdüğü bağlantılar? Bu yazıda, eBPF kullanarak kernel seviyesinde canary sağlığını izleyen ve sorun çıktığında otomatik rollback tetikleyen bir sistemin nasıl kurulduğunu anlatıyorum.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pTlQaC0IlU5QSBRL5h7XOg.png" /><figcaption>Topoloji</figcaption></figure><p>Canary deployment, yeni bir uygulama sürümünü kullanıcıların küçük bir yüzdesine (%10–20) açıp, sorun yoksa kademeli olarak genişletme stratejisidir. Sorun çıktığında ise hızla geri dönülür.</p><p>Klasik bir canary sisteminde karar verici genellikle şunlara bakar:</p><ul><li>HTTP 5xx oranı</li><li>Yanıt gecikmesi (latency)</li><li>Uygulama logları</li></ul><p>Ama bunların hepsi uygulama katmanındaki metrikler. Bir paketin TCP düzeyinde drop edildiğini, SYN paketinin kernelden geri döndüğünü bu metrikler göstermez.</p><p>İşte bu kör noktayı yok etmek için <strong>eBPF (Extended Berkeley Packet Filter)</strong> teknolojisi devreye giriyor. eBPF, Linux çekirdeğine küçük ve güvenli programlar yükleyerek; uygulama koduna dokunmadan ve sıfıra yakın overhead ile ağ trafiğini, sistem çağrılarını ve kernel davranışlarını gerçek zamanlı izlemenizi sağlar.</p><p>Kubernetes kümesinde C veya Go kodları yazmakla uğraşmadan, pratik bir şekilde nasıl kullanabiliriz? Cevap: <strong>Inspector Gadget</strong>.</p><p>Bir CNCF projesi olan Inspector Gadget, eBPF programlarını “gadget” adı verilen hazır araçlar olarak paketleyerek container seviyesinde hata ayıklamamızı (debugging) sağlıyor.</p><h4>Argo Rollout &amp; Inspektor Gadger Kullanarak Problem Tespiti ve Otomatik Rollout</h4><p>Bu bölümde; <strong>Inspektor Gadget</strong>’ın tcpdrop yeteneğini kullanarak canary pod’larındaki problemleri nasıl anlık olarak yakaladığımızı ve bu veriyi <strong>Argo Rollouts</strong> ile paylaşarak sorunlu canary sürümünü (v2) saniyeler içinde nasıl otomatik olarak geri çektiğimizi (automated rollback) bir vaka analizi (POC) ile anlatıyorum.</p><p><strong>Inspector Gadget `kubectl gadget deploy`</strong> komutuyla bir DaemonSet olarak her node’a kurulur. Bu DaemonSet, üzerinde çalıştığı node’un kernel’ına eBPF programları yükleyebilir.</p><p>TCP drop’larını kernel seviyesinde gerçek zamanlı izlemek için bu komut kullanılır.</p><pre>kubectl gadget run trace_tcpdrop:latest -n default</pre><p>Bu komut tetiklendiğinde arka planda şu adımlar gerçekleşir:</p><ol><li>Inspektor Gadget, ilgili node’un kernel space’ine bir <strong>eBPF kprobe</strong> (kernel probe) yükler.</li><li>Bu kprobe, Linux çekirdeğinin ağ paketlerini serbest bıraktığı/imha ettiği ana fonksiyon olan kfree_skb yapısına kanca (hook) atar.</li><li>Kümede gerçekleşen her TCP drop olayında; veri <strong>Kernel Space → eBPF Maps → User Space</strong> hattı üzerinden neredeyse sıfır gecikmeyle terminalimize akar.</li></ol><pre>NODE                    NAMESPACE   POD                         NETNS      PID  COMM  REASON<br>kind-control-plane      default     canary-demo-d6689-kv5q9     4026532678 1234 nginx NO_SOCKET</pre><p>Inspektor Gadget’ın terminalde canlı log göstermesi sorun çözme (debugging) anında kullanışlıdır, ancak bunu otomatik bir rollback mekanizmasına bağlamak için veriyi metrikleştirmemiz gerekir.</p><p>Inspektor Gadget, arka planda aktif çalışan tüm gadget’lar için <strong>port </strong><strong>2223</strong> üzerinden yerleşik (native) Prometheus formatında metrik yayınlar. Bu metrikleri kümemizdeki kube-prometheus-stack mimarisinin otomatik toplayabilmesi (scrape) için bir Service ve ServiceMonitor kaynağı tanımlıyoruz:</p><pre># manifests/gadget-servicemonitor.yaml<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: inspektor-gadget-metrics<br>  namespace: gadget<br>  labels:<br>    k8s-app: gadget<br>spec:<br>  selector:<br>    k8s-app: gadget<br>  ports:<br>  - name: metrics<br>    port: 2223<br>    targetPort: 2223<br>---<br>apiVersion: monitoring.coreos.com/v1<br>kind: ServiceMonitor<br>metadata:<br>  name: inspektor-gadget<br>  namespace: monitoring<br>  labels:<br>    release: kube-prometheus-stack<br>spec:<br>  namespaceSelector:<br>    matchNames:<br>    - gadget<br>  selector:<br>    matchLabels:<br>      k8s-app: gadget<br>  endpoints:<br>  - port: metrics<br>    interval: 15s</pre><p>Argo Rollouts, standart Kubernetes Deployment nesnesinin yerine geçen ve bize çok daha gelişmiş trafik yönetim yetenekleri sunan bir Rollout CRD&#39;si (Custom Resource Definition) tanımlar.</p><pre># manifests/rollout-app.yaml<br>apiVersion: argoproj.io/v1alpha1<br>kind: Rollout<br>metadata:<br>  name: canary-demo<br>  namespace: default<br>spec:<br>  replicas: 5<br>  selector:<br>    matchLabels:<br>      app: canary-demo<br>  template:<br>    metadata:<br>      labels:<br>        app: canary-demo<br>        version: v1<br>    spec:<br>      containers:<br>        - name: canary-demo<br>          image: nginx:1.25-alpine<br>          ports:<br>            - containerPort: 80<br>          resources:<br>            requests:<br>              cpu: 10m<br>              memory: 32Mi<br>            limits:<br>              cpu: 100m<br>              memory: 64Mi<br>  strategy:<br>    canary:<br>      analysis:<br>        templates:<br>        - templateName: ebpf-canary-health<br>        startingStep: 1<br>      steps:<br>        - setWeight: 20<br>        - pause: {}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Npk6pWgiYgRSZounpg2hUw.png" /><figcaption>Rollout Status</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/1*MAcmsRoGOraw2tAX9Qu_Pg.png" /><figcaption>Pod Status</figcaption></figure><p>strategy.canary bloğundaki kritik kontrol noktaları:</p><ul><li><strong>setWeight: 20:</strong> Yeni sürüm (v2) çıktığı an, trafiğin tam olarak %20&#39;si bu canary pod&#39;larına yönlendirilir.</li><li><strong>pause: {}:</strong> Herhangi bir zaman sınırı olmaksızın süresiz bekleme modudur. Sistem ya manuel bir onay ya da arkada çalışan otomatik analizörün (AnalysisTemplate) kararını bekler.</li><li><strong>analysis.templates:</strong> Canary süreci başladığı ilk andan itibaren arka planda hangi sağlık kontrol mekanizmasının çalışacağını söyler.</li></ul><h4>İki Sistem Nasıl Haberleşiyor?</h4><p>Inspektor Gadget kernel katmanında TCP drop’larını yakalıyor. Argo Rollouts ise yukarıda deployment’ı yönetiyor. Peki, aralarında doğrudan bir entegrasyon API’si var mı?</p><p><strong>Cevap: Hayır.</strong> İkisi tamamen bağımsız çalışan sistemler. Linux Kernel’ında drop edilen her paket, uygulama katmanında bir curl timeout veya connection refused fırlatılmasına neden olur.</p><p>Bu iki dünyayı birbirine bağlayan sihirli köprü <strong>AnalysisTemplate</strong> kaynağıdır. Bu şablon, Argo Rollouts&#39;a periyodik olarak çalıştıracağı bir Kubernetes Job prototipi sunar:</p><pre># manifests/analysis-template.yaml<br>apiVersion: argoproj.io/v1alpha1<br>kind: AnalysisTemplate<br>metadata:<br>  name: ebpf-canary-health<br>  namespace: default<br>spec:<br>  metrics:<br>  - name: canary-pod-reachability<br>    interval: 30s<br>    failureLimit: 1<br>    successCondition: result == &quot;ok&quot;<br>    provider:<br>      job:<br>        spec:<br>          backoffLimit: 0<br>          template:<br>            spec:<br>              serviceAccountName: ebpf-analysis-sa<br>              restartPolicy: Never<br>              containers:<br>              - name: probe<br>                image: bitnami/kubectl:latest<br>                command: [&quot;/bin/bash&quot;, &quot;-c&quot;]<br>                args:<br>                - |<br>                  PODS=$(kubectl get pods -n default -l version=v2 \<br>                    -o jsonpath=&#39;{.items[*].status.podIP}&#39; 2&gt;/dev/null)<br><br>                  if [ -z &quot;$PODS&quot; ]; then<br>                    echo &quot;no-canary-pods&quot;<br>                    echo &quot;ok&quot;<br>                    exit 0<br>                  fi<br><br>                  FAIL=0<br>                  for IP in $PODS; do<br>                    if curl -sf --max-time 4 &quot;http://$IP:80&quot; &gt; /dev/null 2&gt;&amp;1; then<br>                      echo &quot;OK: $IP reachable&quot;<br>                    else<br>                      echo &quot;DROP detected: $IP unreachable&quot;<br>                      FAIL=$((FAIL+1))<br>                    fi<br>                  done<br><br>                  if [ &quot;$FAIL&quot; -gt 0 ]; then<br>                    echo &quot;FAIL: $FAIL canary pod(s) unreachable&quot;<br>                    exit 1<br>                  fi<br>                  echo &quot;ok&quot;</pre><h4>Bu Otomasyon Arka Planda Ne Yapıyor?</h4><p>Canary deployment askıda beklerken (pause adımında), her <strong>30 saniyede bir</strong> şu çarklar döner:</p><ol><li>Argo Rollouts, bitnami/kubectl imajını kullanan geçici bir kontrol pod&#39;u (K8s Job) ayağa kaldırır.</li><li>Bu pod, kubectl get pods -l version=v2 komutunu işleterek sahada canlı olan tüm <strong>Canary (v2)</strong> pod&#39;larının IP adreslerini toplar.</li><li>Elde ettiği IP’lere tek tek curl -sf --max-time 4 komutuyla hızlı sağlık istekleri gönderir.</li><li>Eğer paket kernel seviyesinde drop yiyorsa (Inspektor Gadget’ın trace_tcpdrop üzerinde logladığı kriz anı), curl komutu başarısız olur ve betik <strong>exit 1</strong> fırlatır.</li><li>Job’ın hata vermesi üzerine AnalysisRun bu adımı başarısız sayar. Tanımladığımız <strong>failureLimit: 1</strong> sınırı aşıldığı an analiz Failed durumuna çekilir.</li><li>Argo Rollouts bu başarısızlığı görür görmez canary sürecini anında iptal eder (Abort) ve trafiği saniyeler içinde eski stabil sürüme (v1) geri çeker. <strong>Mükemmel bir otomatik rollback</strong></li></ol><p>Analiz Job’ının içerisinden kubectl get pods komutunu güvenle koşturabilmek için Kubernetes API sunucusuna kendimizi tanıtmamız gerekir. SRE dünyasının en önemli kuralı olan <strong>&quot;En Az Ayrıcalık İlkesi&quot; (Principle of Least Privilege)</strong> uyarınca sadece okuma yetkisine sahip bir RBAC profili kurguluyoruz:</p><pre># manifests/ebpf-analysis-rbac.yaml<br>apiVersion: v1<br>kind: ServiceAccount<br>metadata:<br>  name: ebpf-analysis-sa<br>  namespace: default<br>---<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: ClusterRole<br>metadata:<br>  name: ebpf-analysis-role<br>rules:<br>- apiGroups: [&quot;&quot;]<br>  resources: [&quot;pods&quot;]<br>  verbs: [&quot;get&quot;, &quot;list&quot;]<br>---<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: ClusterRoleBinding<br>metadata:<br>  name: ebpf-analysis-rolebinding<br>subjects:<br>- kind: ServiceAccount<br>  name: ebpf-analysis-sa<br>  namespace: default<br>roleRef:<br>  apiGroup: rbac.authorization.k8s.io<br>  kind: ClusterRole<br>  name: ebpf-analysis-role</pre><p>İnşa ettiğimiz eBPF ve Argo Rollouts koruma kalkanının gerçekten çalışıp çalışmadığını test etmek için üretim ortamında kontrollü bir başarısızlık (fault injection) simüle etmemiz gerekiyor.</p><pre># scripts/trigger_canary_failure.sh<br>#!/bin/bash<br>set -e<br><br>echo &quot;🚀 Step 1: Patching Rollout to trigger v2 deployment...&quot;<br>kubectl patch rollout canary-demo -n default --type=&#39;json&#39; -p=&#39;[<br>  {&quot;op&quot;:&quot;replace&quot;,&quot;path&quot;:&quot;/spec/template/spec/containers/0/image&quot;,&quot;value&quot;:&quot;nginx:1.26-alpine&quot;},<br>  {&quot;op&quot;:&quot;replace&quot;,&quot;path&quot;:&quot;/spec/template/metadata/labels/version&quot;,&quot;value&quot;:&quot;v2&quot;}<br>]&#39;<br><br>echo &quot;⏳ Step 2: Waiting for Canary (v2) pod to become Ready...&quot;<br>kubectl wait --for=condition=ready pod -l version=v2 -n default --timeout=120s<br><br>echo &quot;💥 Step 3: Injecting Chaos! Modifying NGINX config inside v2 pod...&quot;<br>V2_POD=$(kubectl get pods -n default -l version=v2 -o jsonpath=&#39;{.items[0].metadata.name}&#39;)<br><br>kubectl exec -n default &quot;$V2_POD&quot; -- sh -c \<br>  &#39;printf &quot;server { listen 80; return 503; }&quot; &gt; /etc/nginx/conf.d/default.conf &amp;&amp; nginx -s reload&#39;<br><br>echo &quot;✅ Chaos injected successfully. Watch the automated rollback unfold!&quot;</pre><p>Arka planda sürekli olarak endpoint’e istek atan bir trafik scripti çalıştırıyoruz.</p><pre># scripts/generate_traffic.sh<br>#!/bin/bash<br>URL=&quot;http://localhost:30000&quot; # Local NodePort endpoint<br>SUCCESS=0<br>FAIL=0<br><br>echo &quot;🚗 Starting continuous traffic stream to $URL...&quot;<br>while true; do<br>  STATUS=$(curl -s -o /dev/null -w &quot;%{http_code}&quot; --max-time 2 &quot;$URL&quot;)<br>  <br>  if [[ &quot;$STATUS&quot; == &quot;200&quot; ]]; then<br>    SUCCESS=$((SUCCESS+1))<br>    printf &quot;[%s] ✅ HTTP %s (Success: %d | Errors: %d)\n&quot; &quot;$(date &#39;+%H:%M:%S&#39;)&quot; &quot;$STATUS&quot; &quot;$SUCCESS&quot; &quot;$FAIL&quot;<br>  else<br>    FAIL=$((FAIL+1))<br>    printf &quot;[%s] 🚨 HTTP %s ← Canary failure detected! (Success: %d | Errors: %d)\n&quot; &quot;$(date &#39;+%H:%M:%S&#39;)&quot; &quot;$STATUS&quot; &quot;$SUCCESS&quot; &quot;$FAIL&quot;<br>  fi<br>  <br>  sleep 0.3<br>done</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XvJjOY4VHDa3lBSg7df3lg.png" /><figcaption>Deployment Başlıyor</figcaption></figure><ul><li><strong>Trafik Başlar:</strong> Trafik üreticinin gönderdiği isteklerin <strong>%20&#39;si</strong> yeni v2 poduna gider HTTP 503 hatası üretir.</li><li><strong>Analysis Run:</strong> Periyodik olarak (her 30 saniyede bir) uyanan AnalysisRun kontrol podu (Job), canary podunun IP&#39;sine (curl) gönderir.</li><li><strong>Tarama Sonucu:</strong> Taramadan 503 yanıtı döndüğü an, betik exit 1 fırlatır ve ilk analiz adımı başarısız (Failed) olarak işaretlenir.</li><li><strong>Argo Rollouts Devreye Girer:</strong> Belirlediğimiz <strong>failureLimit: 1</strong> sınırı aşıldığı an (ikinci periyotta da hata alındığında), Argo Rollouts tüm deployment sürecini durdurur (<strong>Abort</strong>).</li><li><strong>Abort:</strong> Saniyeler içinde v2 (Canary) podları otomatik olarak sıfıra (scale-down) indirilir. Tüm canlı trafik, hiçbir insan müdahalesi olmadan tamamen güvenli olan v1 ReplicaSet&#39;ine geri yönlendirilir.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ChO9mBy0MXQg53o153cdOA.png" /><figcaption>Abort</figcaption></figure><p>Kurduğumuz bu proaktif yapının kurumsal düzeyde anlam kazanması ve ekipler tarafından izlenebilmesi için <strong>New Relic</strong> sistemin merkezi gözlemlenebilirlik (observability) katmanını oluşturuyor. Bu mimaride New Relic, kümemizden <strong>üç farklı veri hattı</strong> üzerinden besleniyor:</p><ul><li><strong>1. K8s Altyapı Metrikleri (</strong><strong>nri-kubernetes):</strong> Pod durumları, node seviyesindeki CPU/Bellek tüketimleri ve container kaynak limitleri gibi temel metrikler, kümede koşan standart New Relic Kubernetes agent&#39;ı tarafından otomatik olarak toplanır.</li><li><strong>2. eBPF Metrikleri (</strong><strong>newrelic-prometheus-agent):</strong> Bir önceki adımda bahsettiğimiz, Inspektor Gadget&#39;ın port 2223 üzerinde açtığı Prometheus endpoint&#39;ini anlık olarak tarar (scrape). Gadget aktif olarak çalışırken üretilen tüm inspektor_gadget_* ön ekine (prefix) sahip derin kernel metrikleri bu hat üzerinden New Relic&#39;e akar.</li><li><strong>3. eBPF Gözlemlenebilirlik Ajanı (</strong><strong>nr-ebpf-agent):</strong> Pixie tabanlı çalışan bu ajan, network katmanının çok daha derinlerine inerek uygulama izleme verileri sağlar.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WGLUhGZS6-dWZIkVBlWisQ.png" /><figcaption>Canary Dashboard</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zf9arCn-_SCa3crX52ITEw.png" /><figcaption>eBPF Dashboard</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RLncSSS7QACq1Z2lb71bdA.png" /><figcaption>Infra Metrics</figcaption></figure><p>Bu Proof of Concept (POC) çalışmasında inşa ettiğimiz sistem; endüstrideki klasik, yavaş ve yüzeysel kalan HTTP durum kodu bazlı canary analizlerine karşı güçlü bir <strong>altyapısal alternatif</strong> sunuyor.</p><p>Geleneksel yöntemler bir facianın kullanıcıya yansımasını (HTTP 5xx) beklerken, <strong>Inspektor Gadget</strong> ve <strong>eBPF</strong> programları sayesinde ağ seviyesindeki kırılmaları ve TCP drop olaylarını daha ilk milisaniyede çekirdek (kernel) katmanında yakalayabiliyoruz.</p><h4>Kaynaklar</h4><ul><li><a href="https://www.inspektor-gadget.io/docs/">Inspektor Gadget Documentation</a></li><li><a href="https://argo-rollouts.readthedocs.io/en/stable/features/analysis/">Argo Rollouts Progressive Delivery Architecture</a></li><li><a href="https://ebpf.io/what-is-ebpf/">eBPF.io — An Introduction to Extended BFF</a></li><li><a href="https://kubernetes.io/docs/reference/networking/virtual-ips/">Kubernetes kube-proxy iptables Mode Deep Dive</a></li><li><a href="https://docs.newrelic.com/docs/kubernetes-pixie/kubernetes-integration/get-started/introduction-kubernetes-integration/">New Relic Kubernetes &amp; Pixie Integration Guide</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=04ca9a7cd7ae" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SRE Teams’ Biggest Nightmare: How to Solve “Alert Fatigue” with Keep (keephq.dev)]]></title>
            <link>https://towardsdev.com/sre-teams-biggest-nightmare-how-to-solve-alert-fatigue-with-keep-keephq-dev-7480f42dd2ec?source=rss-2d45040d93b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/7480f42dd2ec</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[sre]]></category>
            <dc:creator><![CDATA[Kaan Berk ÖZBEK]]></dc:creator>
            <pubDate>Sat, 27 Jun 2026 16:58:43 GMT</pubDate>
            <atom:updated>2026-07-01T09:23:31.978Z</atom:updated>
            <content:encoded><![CDATA[<p>It’s 3:14 AM on a Tuesday. Your phone lights up on the nightstand. Then it buzzes again. And again. By the time you rub the sleep from your eyes, your Slack #alerts-production channel is rolling with a relentless waterfall of notifications.</p><p>This is <strong>Alert Fatigue</strong>, and it is the single biggest silent killer of engineering morale, operational efficiency, and system uptime.</p><p>According to industry data, SRE teams waste up to 40% of their incident triage time simply filtering out redundant noise. When everything is screaming, nothing is heard. Eventually, engineers start ignoring the noise and that’s exactly when a critical, business-ending outage slips through the cracks.</p><p>So, how do we fix a problem caused by having too many tools telling us too many things? We don’t need another dashboard. We need a unified orchestration layer.</p><p>Enter <strong>Keep (keephq.dev)</strong>, the open-source platform designed to put an end to alert chaos.</p><h4>Multi-Tool Fragmented Context</h4><p>In a modern cloud-native architecture, infrastructure is rarely monitored by a single tool. Your frontend might rely on Grafana, your backend microservices are instrumented with Datadog, your databases trigger CloudWatch alarms, and your security team tracks anomalies in Elastic.</p><p>When a core component fails, every single one of these vendors throws an independent tantrum. They don’t talk to each other. They don’t know about each other. They just spam your PagerDuty or Slack.</p><p>To solve this without breaking the bank on enterprise AIOps platforms like BigPanda or PagerDuty’s premium tiers, the open-source community needed a vendor-agnostic tool. That is exactly what Keep delivers.</p><h3>How Keep Eliminates the Noise (The 3 Pillars)</h3><h4>1. Unified Ingestion (The Single Pane of Glass)</h4><p>Instead of forcing your engineers to jump between four different tabs during an active incident, Keep provides 90+ out-of-the-box integrations. Whether an alert comes from Prometheus, Dynatrace, New Relic, or a custom internal script, Keep ingests the raw payload and normalizes it into a single, unified format.</p><h4>2. Alerts-as-Code (Programmable Workflows)</h4><p>This is where Keep completely outshines traditional alert routers. It allows you to manage your alerting workflows using simple, declarative <strong>YAML files</strong> — just like GitHub Actions.</p><p>You can program Keep to say: <em>“If an alert comes from Datadog about </em><em>payment-service latency, automatically query Kubernetes via API to see if a deployment just happened, bundle that context together, and only alert the team if the pod hasn&#39;t restarted.”</em></p><blockquote><strong><em>Example of a Keep Workflow:</em></strong><em> It allows you to enrich, deduplicate, and throttle alerts before they ever reach a human being, turning raw data into actionable intelligence.</em></blockquote><h4>3. Smart Correlation and Deduplication</h4><p>Instead of receiving 50 separate messages for 50 failing pods in the same replica set, Keep deduplicates identical or highly related events using sliding time windows and tag matching.</p><p>Instead of an absolute mess, your on-call engineer receives a single, clean incident story.</p><h3>Let’s Deep Dive Into Keep</h3><h4>Putting It to the Test: My Hands-On Proof of Concept (POC)</h4><p>To see if Keep could actually walk the walk, I decided to build a local <strong>Proof of Concept (POC)</strong> environment. I set up a simulated multi-vendor monitoring stack, intentionally plugging in <strong>Grafana Cloud</strong>, <strong>Prometheus</strong>, and <strong>New Relic</strong> into a Python-based core microservice (keep-poc) that communicates with a downstream demo-application.</p><p>I then triggered a simulated memory leak and service disruption to create an intentional alert storm. Here is a look under the hood of how the platform handled the triage, captured directly from my live POC deployment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3YnOmEayGUFTAkFnb5TiMQ.png" /><figcaption>Dashboard Page</figcaption></figure><p>To measure exactly how efficient Keep is at reducing alert fatigue, I built a custom monitoring interface using the platform’s <strong>Dashboards</strong> feature. The metrics captured during the simulated outage provide clear, data-driven proof of how raw noise is transformed into operational intelligence.</p><p>As visualized in <strong>image,</strong> the metrics tell a compelling story:</p><ul><li><strong>The Alert Storm vs. Incident Squeezing:</strong> Looking at the <strong>Alerts/Day</strong> graph, you can see the exact moment the simulated crisis hit, causing an immediate vertical spike of over 120+ raw, chaotic alerts. However, looking directly above it at the <strong>Incidents</strong> graph, Keep’s correlation engine instantly intercepted the storm, compressing those 120+ frantic notifications into a peak of just 5 highly structured, manageable incidents.</li><li><strong>Granular Alert Quality Tracking:</strong> The dashboard breaks down the exact performance of my multi-vendor setup in the <strong>Alert Quality</strong> ledger:</li><li>Grafana Cloud (kaanbozbek) dispatched 5 raw telemetry events, with a <strong>60.00% correlation rate</strong> directly tied into active incidents.</li><li>Prometheus Local similarly fired 5 alerts, matching the <strong>60.00% correlation efficiency</strong>, proving that cross-vendor alerts were being mapped to the exact same root causes simultaneously.</li><li>New Relic HomeLab injected a baseline anomaly alert, which remained uncorrelated (0.00%) as intended, showing that the engine doesn&#39;t just blindly group everything—it safely isolates independent events.</li></ul><p>This metrics dashboard serves as the ultimate report card for the POC. It proves that instead of throwing more dashboards at engineers, we can use a centralized engine to actively filter out the noise before it drains human cognitive capacity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YNQS1u1VN31HWERGBSDMDw.png" /><figcaption>Incident’s Page</figcaption></figure><blockquote><strong>The Incident Command Center:</strong> Instead of drowning in dozens of scattered notifications, Keep aggregates correlated alerts into distinct “Incidents.” This view provides a high-level executive summary of active infrastructure crises, showing involved services, current firing status, assignee tracking, and the underlying alert count at a glance.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xqK7xbhQbr63HTC4_Y1auw.png" /><figcaption>Feed Page</figcaption></figure><blockquote><strong>Unified Real-Time Feed:</strong> A single pane of glass ingesting raw telemetry events from multi-vendor stacks.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rqkqas75X878hHbwIFqbhA.png" /><figcaption>Deduplication Rules Page</figcaption></figure><blockquote><strong>Intelligent Deduplication Engine:</strong> Alert storms happen when a single event evaluates repeatedly over a short window. Keep dynamically calculates a “Dedup Ratio” by hashing configurations like name + severity. As seen in the metrics, it effortlessly wipes out over 95% of redundant alert noise before it ever hits a human engineer.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y9fpeTKWsFT4g68OKFpvgA.png" /><figcaption>Correlations Page</figcaption></figure><blockquote><strong>Flexible Correlation Logic:</strong> Grouping strategies aren’t hardcoded. Engineers can define custom thresholds and rule semantics such as compiling events based strictly on boundaries like severity == &quot;critical&quot; or logical application names to safely isolate cascading blast radiuses.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w81r5PtAzJ6Cw0O5SBEa3A.png" /><figcaption>Workflows Page</figcaption></figure><blockquote>Shifting from passive observing to active self-healing. Keep enables engineers to write “Alerts-as-Code” automation workflows. When a critical issue triggers, the platform doesn’t just send a message; it can automatically enrich data, trigger webhooks, open Jira tickets, or execute remediation scripts.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KQiKMrvZqttIpX_j6pmUUg.png" /><figcaption>Topology Page</figcaption></figure><blockquote><strong>Visual Environment Topology:</strong> Tracing blast radiuses requires clear architectural visibility. Keep charts an interactive dependency graph maps how monitoring providers feed into specific internal proxy applications, which in turn interface with core user-facing demo applications</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5ZOYviTRr8CG7CXd-g2Y-Q.png" /><figcaption>Mapping Page</figcaption></figure><blockquote><strong>Contextual Metadata Enrichment:</strong> A raw infrastructure alert is useless without context. The Mapping engine auto-injects critical metadata fields — like team ownership, runbook URLs, environment tags, and active on-call assignments straight into incoming payloads via dynamic backend lookup tables.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MHzulj7p-_1bA84ouY76_A.png" /><figcaption>Extractions Page</figcaption></figure><blockquote><strong>Regex-Driven Attribute Extraction:</strong> Telemetry formats vary wildly. Keep solves unstructured payload differences by deploying localized Regex extractions. This process intelligently extracts environment tiers (prod, staging) or alert signatures straight out of raw text blocks to populate searchable tags.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kg0fJJ-YXn_m5Y9gnDRRvg.png" /><figcaption>Maintenance Window Page</figcaption></figure><blockquote><strong>Dynamic Maintenance Windows (Silencing Scheduled Noise):</strong> Infrastructure upgrades, database migrations, and scheduled code deployments shouldn’t trigger false alarms or wake up on-call engineers. Keep’s Maintenance Windows feature allows teams to effortlessly schedule suppression rules. By leveraging granular CEL expressions and precise scheduling, you can automatically muzzle alerts from specific sources during known maintenance intervals, ensuring your main incident feed stays clean and high-fidelity</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PjmqFGN67nI9e-9wq9fiqw.png" /><figcaption>Providers Page</figcaption></figure><blockquote><strong>Multi-Vendor Provider Ecosystem (Zero-Friction Ingestion):</strong> The power of a unified alerting system lies entirely in its connectivity. Keep functions as a universal translator across your entire observability estate. With native support for industry giants — ranging from Prometheus and Grafana to New Relic, Dynatrace, and CloudWatch connecting a new ingestion pipeline is a zero-code process that takes only a few clicks, tearing down data silos instantly.</blockquote><h3>Nice to Know About Keep</h3><p><strong>AI Features:</strong> Keep features a dedicated <strong>AI Plugins</strong> engine designed to automate root-cause analysis (RCA). Instead of replacing deterministic correlation, Keep utilizes LLMs (supporting OpenAI, Anthropic, or local Ollama deployments) asynchronously. When an incident triggers, the AI parses the consolidated payload to generate a human-readable executive summary and suggests actionable troubleshooting steps based on historical runbooks.</p><p><strong>Open-Source vs. Enterprise:</strong> Keep operates on an open-core model. The core platform including 90+ provider integrations, deduplication pipelines, topology mapping, and workflow automations is <strong>100% open-source and free forever</strong>. The paid <strong>Enterprise/Cloud tier</strong> introduces hosted SaaS convenience, advanced RBAC, SAML/SSO compliance, audit logging, and 24/7 enterprise SLA support.</p><p><strong>Deployment Modes:</strong> Depending on your infrastructure and data privacy policies, Keep can be deployed in three ways:</p><ol><li><strong>Docker Compose:</strong> Ideal for local sandbox testing, running the entire stack via a single command in under 5 minutes.</li><li><strong>Kubernetes (Helm Charts):</strong> The standard choice for highly available, scalable enterprise production environments.</li><li><strong>Keep Cloud (SaaS):</strong> A fully managed, zero-maintenance version for teams who want to start correlating alerts instantly without managing underlying servers.</li></ol><h3>Quick Market Comparison</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D4fBddA4WkGeCcHYpHLBeg.png" /><figcaption>Comparison Table</figcaption></figure><h3>Thank you for reading!</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7480f42dd2ec" width="1" height="1" alt=""><hr><p><a href="https://towardsdev.com/sre-teams-biggest-nightmare-how-to-solve-alert-fatigue-with-keep-keephq-dev-7480f42dd2ec">SRE Teams’ Biggest Nightmare: How to Solve “Alert Fatigue” with Keep (keephq.dev)</a> was originally published in <a href="https://towardsdev.com/">Towards Dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Keploy for API Testing: SaaS & On-Premise]]></title>
            <link>https://blog.stackademic.com/keploy-for-api-testing-saas-on-premise-451b56eb3c48?source=rss-2d45040d93b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/451b56eb3c48</guid>
            <category><![CDATA[postman]]></category>
            <category><![CDATA[keploy]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[api-testing]]></category>
            <dc:creator><![CDATA[Kaan Berk ÖZBEK]]></dc:creator>
            <pubDate>Mon, 29 Dec 2025 22:07:44 GMT</pubDate>
            <atom:updated>2026-01-17T08:46:48.891Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article, I have analyzed two different consumption models for Keploy.</p><ul><li><strong>Keploy SaaS</strong> → UI-driven API test generation</li><li><strong>Keploy On-Premise (CLI + Docker)</strong> → traffic-based test recording with dependency capture</li></ul><h3>Keploy SaaS</h3><p>In the SaaS model, Keploy does not “record” your application.<br>Instead, it <strong>analyzes the API surface and response behavior</strong> to synthesize test cases automatically.</p><p>At a high level, Keploy SaaS answers one question:</p><blockquote>“If this endpoint exists, what are the meaningful ways it can be exercised?”</blockquote><p>Sample: httpbin</p><h4>How Tests Are Generated?</h4><p>Keploy SaaS works by combining three signals:</p><p><strong>Request structure</strong></p><ul><li>HTTP method (GET, POST, PUT, PATCH, DELETE)</li><li>Query parameters</li><li>Headers</li><li>Body shape (JSON, form, empty)</li></ul><p><strong>Observed responses</strong></p><ul><li>Status codes</li><li>Response schema</li><li>Header behavior</li><li>Content type and encoding</li></ul><p><strong>Protocol-level heuristics</strong></p><ul><li>Common REST patterns</li><li>Expected error scenarios</li><li>Header-driven behavior (e.g. compression, auth, content negotiation)</li></ul><h4>How to Run Tests?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/1*n-A4OduUOEYfJKAzD1tQ7g.png" /><figcaption>URL Added</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/1*KTnXRLLoj3-g62J8LVV6hA.png" /><figcaption>Input Added</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/1*rQLppSrW6-mQujIOUWQadA.png" /><figcaption>Test Generation</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/1*1OfCyBVV-3mPNxet0uGAjw.png" /><figcaption>Test Generated</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/1*LRZCYU8xOA0yDJeoUN-QxA.png" /><figcaption>Results</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/1*lAg6U4V3Czvbr8VNE4pE4g.png" /><figcaption>Results — 2</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0RQs7Rc-kRY-iWD8AkHErg.png" /><figcaption>Dashboard</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WCWXQv0vDpeFTo2kV8UvIg.png" /><figcaption>Schema</figcaption></figure><h4>What Kind of Tests Get Auto-Generated?</h4><p>Typically, SaaS-generated tests fall into these categories:</p><p><strong>Positive tests</strong></p><ul><li>Valid request → expected 2xx response</li></ul><p><strong>Negative tests</strong></p><ul><li>Missing or malformed input</li><li>4xx responses</li></ul><p><strong>Status-code–focused tests</strong></p><ul><li>Explicit checks for 404, 500, etc.</li></ul><p><strong>Header behavior tests</strong></p><ul><li>User-Agent</li><li>Accept-Encoding</li><li>Content Type</li></ul><p><strong>Method-specific tests</strong></p><ul><li>Same endpoint exercised with different HTTP verbs</li></ul><p><strong>Schema validation tests</strong></p><ul><li>Response body matches inferred JSON schema</li></ul><h3>Results</h3><p><strong>Strengths</strong></p><ul><li>Extremely fast test generation</li><li>No setup required</li><li>Great for API exploration and onboarding</li><li>Useful for early contract validation</li></ul><p><strong>Limitations</strong></p><ul><li>No real traffic capture</li><li>No database or external dependency mocking</li><li>Not state-aware</li><li>Not suitable for CI regression testing</li></ul><p>In short:</p><blockquote><strong><em>SaaS = an intelligent test generator</em></strong><em><br> </em><strong><em>Not a record-and-replay system</em></strong></blockquote><h3>Keploy On-Premise</h3><p>Keploy’s <strong>on-premise mode</strong> is fundamentally different from UI-based or specification-driven testing tools. Instead of generating tests from assumptions or schemas, it builds tests from <strong>real application behavior</strong>, observed at runtime.</p><p>Keploy On-Premise answers a different question than most API testing tools:</p><blockquote>“Did this application behave the same way as before, under the same conditions?</blockquote><p>Sample: <a href="https://keploy.io/docs/quickstart/samples-java/#instructions-for-starting-using-docker">https://keploy.io/docs/quickstart/samples-java/#instructions-for-starting-using-docker</a></p><p>I have deployed the sample app, then recorded the response of app (sending curl) and test it.</p><h4>Keploy:</h4><ul><li>Observes <strong>live traffic</strong></li><li>Captures <strong>dependency interactions</strong></li><li>Replays requests in a <strong>deterministic and isolated manner</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/1*S4YjeNetwWv7ufC9EMvOIA.png" /><figcaption>Keploy Installation</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7_mkSnrjBjq74rkEuoee9A.png" /><figcaption>Keploy Record</figcaption></figure><p><strong>While testing I sent the below request as in sample:</strong></p><p><em>curl — location — request POST ‘http://localhost:8080/api/employees&#39; \<br> — header ‘Content-Type: application/json’ \<br> — data-raw ‘{<br> “firstName”: “Myke”,<br> “lastName”: “Tyson”,<br> “email”: “mt@gmail.com”,<br> “timestamp”:1<br>}’</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*emHA4qHf7ZVVt-y2AYnXBg.png" /><figcaption>Keploy Test</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wnke4tJtBqX0QVwIJjMUyw.png" /><figcaption>mocks.yml</figcaption></figure><p>In test.yml, command and response can be seen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yLQEjdp5d33xCNOzvbbhTw.png" /><figcaption>test.yml</figcaption></figure><h3>What does Keploy do?</h3><h4>Traffic Recording</h4><p>In record mode, Keploy runs alongside the application and intercepts:</p><ul><li>Incoming HTTP requests</li><li>Outgoing HTTP calls</li><li>Database queries</li><li>Calls to external services and dependencies</li></ul><p>Rather than only storing request/response pairs, Keploy correlates:</p><ul><li>API calls</li><li>Dependency calls</li><li>Their exact responses</li></ul><p>The result is a set of <strong>behavioral test artifacts</strong> that represent how the system actually behaved in a given environment.</p><p><strong>Dependency Capture and Mocking</strong></p><p>One of the defining features of Keploy On-Premise is <strong>dependency capture</strong>.</p><p>During recording, Keploy captures interactions with:</p><ul><li>Databases</li><li>Caches (e.g. Redis)</li><li>Message brokers</li><li>External HTTP services</li></ul><p>These interactions are stored as mocks and later replayed during testing.<br>This allows Keploy to run tests without relying on live dependencies, while still validating the application logic against real responses.</p><p>In practice, this enables:</p><ul><li>Stable test execution</li><li>Isolation from flaky external systems</li><li>Reproducible failures</li></ul><p><strong>Deterministic Replay (Test Phase)</strong></p><p>During replay, Keploy:</p><ul><li>Replays recorded API requests</li><li>Serves dependency responses from captured mocks</li><li>Compares current behavior with recorded behavior</li></ul><p>Because dependencies are mocked using real captured data, test runs are <strong>deterministic</strong>.</p><p>This is a critical property for CI/CD pipelines, where non-deterministic tests quickly become noise.</p><p><strong>CI/CD Integration</strong></p><p>Keploy On-Premise is designed to run <strong>inside CI/CD pipelines</strong>.</p><p>A typical workflow looks like this:</p><p><strong>Record phase</strong></p><ul><li>Run in a controlled environment (e.g. staging)</li><li>Capture traffic and dependencies</li><li>Commit test artifacts to version control</li></ul><p><strong>Replay phase</strong></p><ul><li>Run in CI before or after deployment</li><li>Use captured mocks instead of live dependencies</li><li>Fail the pipeline on behavioral regressions</li></ul><p>Source: <a href="https://keploy.io/docs/ci-cd/jenkins/">https://keploy.io/docs/ci-cd/</a></p><h4>Example</h4><pre>pipeline {<br>    agent any<br>    stages {<br>        stage(&#39;Install Dependencies&#39;) {<br>            steps {<br>                sh &#39;sudo apt-get update &amp;&amp; sudo apt-get install -y curl kmod linux-headers-generic bpfcc-tools git golang-go&#39;<br>            }<br>        }<br>        stage(&#39;Clone and Setup App&#39;) {<br>            steps {<br>                sh &#39;&#39;&#39;<br>                rm -rf samples-go<br>                git clone &#39;https://github.com/keploy/samples-go&#39;<br>                cd gin-mongo<br>                go mod tidy<br>                &#39;&#39;&#39;<br>            }<br>        }<br>        stage(&#39;Install Keploy&#39;) {<br>            steps {<br>                sh &#39;&#39;&#39;<br>                curl --silent -O -L https://keploy.io/install.sh &amp;&amp; bash install.sh<br>                &#39;&#39;&#39;<br>            }<br>        }<br>        stage(&#39;Prepare eBPF Hooks&#39;) {<br>            steps {<br>                sh &#39;&#39;&#39;<br>                sudo mkdir -p /sys/kernel/debug<br>                sudo mkdir -p /sys/kernel/tracing<br>                sudo mount -t debugfs nodev /sys/kernel/debug || true<br>                sudo mount -t tracefs nodev /sys/kernel/tracing || true<br>                &#39;&#39;&#39;<br>            }<br>        }<br>        stage(&#39;Run Keploy Tests&#39;) {<br>            steps {<br>                sh &#39;&#39;&#39;<br>                cd gin-mongo<br>                sudo -E keploy test -c &quot;go run main.go handler.go&quot; --disableANSI<br>                &#39;&#39;&#39;<br>            }<br>        }<br>    }<br>}</pre><p>This approach provides <strong>true regression protection</strong>, not just contract or schema validation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*c8-7SshynG3FWnLY1MC3ig.png" /><figcaption>Comparison</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fBoHEDFEK0hCu6BxDVB_kQ.png" /><figcaption>Keploy Commands</figcaption></figure><p>Thanks for reading</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=451b56eb3c48" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/keploy-for-api-testing-saas-on-premise-451b56eb3c48">Keploy for API Testing: SaaS &amp; On-Premise</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Best Practices for Writing Dockerfiles]]></title>
            <link>https://blog.stackademic.com/best-practices-for-writing-dockerfiles-b6a08010de57?source=rss-2d45040d93b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/b6a08010de57</guid>
            <category><![CDATA[dockerfiles]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Kaan Berk ÖZBEK]]></dc:creator>
            <pubDate>Thu, 04 Dec 2025 11:14:01 GMT</pubDate>
            <atom:updated>2025-12-08T17:14:23.043Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6E_aKz3rTqpqwgwGVMekqg.png" /><figcaption>Docker Steps</figcaption></figure><p>Dockerfile design needs a clear standard. Below is a collection of practices I use myself, combined with well-established industry standards.</p><h4>Choose the right base image</h4><p>First rule: pick a base image that fits your purpose and is as lightweight as possible.</p><p>For example for an OS: alpine or distroless makes far more sense than ubuntu or debian.</p><p>Version pinning is non-negotiable:</p><pre>FROM node:22-alpine</pre><p>Using latest breaks reproducible builds and introduces surprises in CI/CD.</p><h4>Use Multi-Stage Builds</h4><p>Separating build, test, and runtime phases is one of the biggest advantages of Docker.</p><p>A simple Java example:</p><pre>FROM maven:3.9-eclipse-temurin-21 AS build<br>WORKDIR /app<br>COPY pom.xml .<br>RUN mvn dependency:go-offline<br>COPY . .<br>RUN mvn package -DskipTests<br>FROM eclipse-temurin:21-jre-alpine<br>WORKDIR /app<br>COPY – from=build /app/target/app.jar app.jar<br>ENTRYPOINT [&quot;java&quot;, &quot;-jar&quot;, &quot;app.jar&quot;]</pre><p>The build stage can have too many packages, toolchains, compilers, etc.</p><p>But the final image must haveonly the minimal output needed to run your application.</p><p>This keeps the image small.</p><h4>Order your layers to maximize cache efficiency</h4><p>Docker caching can drastically improve build speed.</p><p>Layers that change rarely should come first, while source code changes should be at the bottom.</p><p>Bad example;</p><pre>COPY . .<br>RUN npm install</pre><p>Good example;</p><pre>COPY package*.json ./<br>RUN npm install – production<br>COPY . .</pre><h4>Combine RUN commands and clean up after package managers</h4><p>Every RUN instruction creates a separate layer.</p><h4>Combine commands and clean up unnecessary files:</h4><pre>RUN apk add – no-cache curl bash</pre><pre>RUN apt-get update \<br>&amp;&amp; apt-get install -y curl bash \<br>&amp;&amp; rm -rf /var/lib/apt/lists/*</pre><h4>Run your container as a non-root user</h4><p>Running as root inside a container is an avoidable security risk.</p><pre>RUN addgroup -S app &amp;&amp; adduser -S app -G app<br>USER app<br>This also avoids issues with Kubernetes/Openshift security policies.<br>Use ENTRYPOINT and CMD correctly<br>ENTRYPOINT: the main process<br>CMD: optional default arguments<br>ENTRYPOINT [&quot;java&quot;, &quot;-jar&quot;, &quot;app.jar&quot;]<br>CMD [&quot; – spring.profiles.active=prod&quot;]</pre><h4>Do not put configuration into the image</h4><p>Configuration must be taken from environment variables or external config providers like ConfigMaps or Secrets in Kubernetes.</p><pre>ENV APP_PORT=8080<br>EXPOSE 8080<br>Never put secrets in a Dockerfile<br>ENV DB_PASSWORD=…</pre><p>Secrets belong in your CI/CD secret store, Vault, or Kubernetes Secrets not inside the image.</p><h4>Add a Healthcheck when appropriate</h4><p>Let your orchestrator understand the application’s health:</p><pre>HEALTHCHECK – interval=30s – timeout=5s – retries=3 \<br>. CMD curl -f http://localhost:8080/health || exit 1</pre><h4>Write logs to stdout/stderr</h4><p>Don’t write logs to files inside the container.</p><p>Every major logging system Loki, ELK, Splunk, Dynatrace, Instana etc. collects logs from stdout/stderr anyway.</p><h4>Add metadata labels</h4><p>Useful for observability, debugging, and CI/CD pipelines:</p><pre>LABEL app.name=&quot;payments-api&quot; \<br>. app.version=&quot;1.3.0&quot; \<br>. app.build_date=&quot;2025–12–04&quot;</pre><p>One container = one responsibility</p><p>Running Nginx, your backend, a scheduler, and a database all inside the same image is poor architecture.</p><p>Each container should have a single purpose; orchestration is handled by the platform, not the image.</p><h4>Bonus 1: Always use .dockerignore</h4><p>Sending your entire repository as build context is one of the biggest anti-patterns.</p><p>A clean .dockerignore speeds up builds and keeps the final image uncluttered:</p><pre>dockerignore<br><br># git <br>.git<br><br># node<br><br>node_modules<br><br>#logs<br><br>target<br>*.log</pre><h4>Bonus 2: Handle PID 1 signals properly (graceful shutdown)</h4><p>If your application runs as PID 1, signals like SIGTERM must be propagated correctly.</p><p>If you wrap the app in a shell script, make sure to use exec:</p><pre>exec java -jar app.jar &quot;$@&quot;</pre><p>This ensures the application shuts down cleanly.</p><h4>Result</h4><p>Choosing the right base image, leveraging multi-stage builds, structuring layers for caching, using non-root users, managing config and secrets correctly – all of it adds up to a clean, predictable, production-ready workflow.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b6a08010de57" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/best-practices-for-writing-dockerfiles-b6a08010de57">Best Practices for Writing Dockerfiles</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Chaos Engineering: Strengthening Systems by Breaking Them on Purpose]]></title>
            <link>https://blog.stackademic.com/chaos-engineering-strengthening-systems-by-breaking-them-on-purpose-d9e5ee9cab9b?source=rss-2d45040d93b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/d9e5ee9cab9b</guid>
            <category><![CDATA[site-reliability-engineer]]></category>
            <category><![CDATA[resilience]]></category>
            <category><![CDATA[chaos-engineering]]></category>
            <dc:creator><![CDATA[Kaan Berk ÖZBEK]]></dc:creator>
            <pubDate>Thu, 09 Oct 2025 11:45:24 GMT</pubDate>
            <atom:updated>2025-10-09T11:45:24.255Z</atom:updated>
            <content:encoded><![CDATA[<p>Perfect systems live only in diagrams. In the real world, systems are messy, unpredictable, and fragile. <strong>Chaos Engineering</strong> embraces that chaos instead of denying it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*wMx_ZyDX0auRI06jqHz4ow@2x.jpeg" /></figure><h3>Why We Break Things on Purpose?</h3><p>Traditional engineering aims to prevent errors.<br>Chaos Engineering flips that logic upside down and asks:</p><blockquote><em>“If failure is inevitable, why not choose when it happens?”</em></blockquote><p>The idea is to expose weaknesses <em>before</em> real incidents strike. Netflix popularized this mindset with its legendary <strong>Simian Army</strong> — a suite of tools like <em>Chaos Monkey</em> that randomly terminate services in production. The mission wasn’t destruction. It was resilience training. A healthy system should survive even when part of it vanishes without warning.</p><h3>A Scientific Experiment, Not Random Destruction</h3><p>Chaos Engineering isn’t reckless; it’s methodical — like a scientific experiment.</p><ol><li><strong>Form a Hypothesis:</strong><br> <em>If one node shuts down, traffic will automatically reroute to another.</em></li><li><strong>Run the Experiment:</strong><br> Intentionally kill a node in a real or production-like environment.</li><li><strong>Observe and Analyze:</strong><br> Did traffic reroute? Did latency spike? Were users affected?</li></ol><p>The point is not to break things for fun — but to collect <strong>data about failure</strong>. Each experiment replaces assumption with evidence and reveals how resilient your system truly is.</p><h3>What to Break Examples</h3><h4>1. Network Chaos</h4><p>Introduce packet loss, latency, or full disconnections to simulate unstable connections.</p><h4>2. Resource Chaos</h4><p>Throttle CPU, memory, or disk resources to reveal performance bottlenecks.</p><h4>3. Dependency Chaos</h4><p>Shut down or slow a dependent service and observe how upstream systems respond.</p><h4>4. Infrastructure Chaos</h4><p>Kill servers or containers unexpectedly and test recovery mechanisms.</p><h3>Real-World Applications</h3><p>Chaos Engineering isn’t just for Netflix-sized infrastructures.<br> It’s relevant anywhere reliability matters.</p><ul><li><strong>Finance:</strong> Testing tolerance during network hiccups in transaction pipelines.</li><li><strong>E-commerce:</strong> Simulating checkout or payment service failures to assess customer impact.</li><li><strong>Telecommunications:</strong> Stress-testing auto-scaling and failover during traffic surges.</li></ul><p>By running controlled chaos, organizations strengthen not just their systems — but their <strong>processes and people</strong> as well.</p><h3>What We Learn from Chaos?</h3><p>Every chaos experiment is a lesson in resilience. It tests not only <strong>software</strong>, but also <strong>human response</strong>. When a node fails, logs matter — but so does the first Slack message sent by your on-call engineer. Chaos reveals how well teams communicate under stress and how clearly they understand recovery procedures.</p><p><em>Resilience isn’t born from comfort. It’s forged in controlled failure.</em></p><p>Netflix captured it perfectly:</p><blockquote><em>“You build confidence by breaking things on purpose.”</em></blockquote><h3>The Future: Intelligent Chaos</h3><p>As AI and automation evolve, Chaos Engineering is becoming smarter. Modern tools like <strong>Gremlin</strong>, <strong>Steadybit</strong>, and <strong>Litmus</strong> can automatically detect weak points and suggest experiments. Instead of asking “What should we break today?”, engineers might soon ask:</p><blockquote><em>“What does the AI think we should test next?”</em></blockquote><p>Chaos, in this sense, becomes <strong>strategic</strong> — an ongoing feedback loop that continuously strengthens system reliability. True resilience comes not from avoiding failure but from <strong>understanding it. </strong>We don’t test chaos to destroy; we do it to evolve. Because systems that ignore chaos are eventually destroyed by it.</p><h3>A message from our Founder</h3><p><strong>Hey, </strong><a href="https://linkedin.com/in/sunilsandhu"><strong>Sunil</strong></a><strong> here.</strong> I wanted to take a moment to thank you for reading until the end and for being a part of this community.</p><p>Did you know that our team run these publications as a volunteer effort to over 3.5m monthly readers? <strong>We don’t receive any funding, we do this to support the community. ❤️</strong></p><p>If you want to show some love, please take a moment to <strong>follow me on </strong><a href="https://linkedin.com/in/sunilsandhu"><strong>LinkedIn</strong></a><strong>, </strong><a href="https://tiktok.com/@messyfounder"><strong>TikTok</strong></a>, <a href="https://instagram.com/sunilsandhu"><strong>Instagram</strong></a>. You can also subscribe to our <a href="https://newsletter.plainenglish.io/"><strong>weekly newsletter</strong></a>.</p><p>And before you go, don’t forget to <strong>clap</strong> and <strong>follow</strong> the writer️!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d9e5ee9cab9b" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/chaos-engineering-strengthening-systems-by-breaking-them-on-purpose-d9e5ee9cab9b">Chaos Engineering: Strengthening Systems by Breaking Them on Purpose</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setting Up Consul Server and Client]]></title>
            <link>https://medium.com/@kaanberkozbek/setting-up-consul-server-and-client-cd0d6c6dc3e6?source=rss-2d45040d93b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/cd0d6c6dc3e6</guid>
            <category><![CDATA[consul]]></category>
            <category><![CDATA[high-availability]]></category>
            <category><![CDATA[etcd]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Kaan Berk ÖZBEK]]></dc:creator>
            <pubDate>Tue, 03 Jun 2025 05:36:53 GMT</pubDate>
            <atom:updated>2025-06-03T05:36:53.772Z</atom:updated>
            <content:encoded><![CDATA[<p>Lately, while working on my PostgreSQL cluster, I wanted to share how I built a <strong>highly available</strong> and <strong>dynamic</strong> architecture using Consul. With Consul, <strong>leader node</strong> management, <strong>service discovery</strong>, and <strong>health checks</strong> become much easier. In this article, I’ll walk through how I set up Consul servers and integrated client nodes like PgCat.</p><p><strong>Note:</strong> I’ve masked sensitive details like IPs and keys as &lt;VARIABLES&gt;. Be sure to replace them in your environment!</p><p>On the server side, I built a 3-node Consul HA cluster. This ensures that if the leader node fails, the remaining nodes automatically elect a new leader.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*twFMQ0XSatgScmI_1YiANQ.png" /><figcaption>Consul — 1</figcaption></figure><pre>#!/bin/bash<br><br>NODE_NAME=&quot;&lt;SERVER_NODE_NAME&gt;&quot;<br>PRIVATE_IP=&quot;&lt;SERVER_PRIVATE_IP&gt;&quot;<br>CONSUL_VERSION=&quot;&lt;CONSUL_VERSION&gt;&quot;<br>CLUSTER_MEMBERS=&quot;&lt;SERVER_IP_1&gt; &lt;SERVER_IP_2&gt; &lt;SERVER_IP_3&gt;&quot;<br><br># Install dependencies<br>apt-get update<br>apt-get install -y unzip curl jq<br><br># Download and install Consul<br>curl -O https://releases.hashicorp.com/consul/${CONSUL_VERSION}/consul_${CONSUL_VERSION}_linux_amd64.zip<br>unzip consul_${CONSUL_VERSION}_linux_amd64.zip<br>mv consul /usr/local/bin/<br>chmod +x /usr/local/bin/consul<br><br># Create Consul user and directories<br>useradd --system --home /etc/consul.d --shell /bin/false consul<br>mkdir --parents /opt/consul /etc/consul.d<br>chown -R consul:consul /opt/consul /etc/consul.d<br><br># Generate gossip encryption key (only once)<br>if [ ! -f /etc/consul.d/gossip.key ]; then<br>  consul keygen &gt; /etc/consul.d/gossip.key<br>fi<br><br># Consul config<br>cat &lt;&lt;EOF &gt; /etc/consul.d/consul.hcl<br>datacenter = &quot;dc1&quot;<br>data_dir = &quot;/opt/consul&quot;<br>encrypt = &quot;$(cat /etc/consul.d/gossip.key)&quot;<br>node_name = &quot;${NODE_NAME}&quot;<br>server = true<br>bind_addr = &quot;${PRIVATE_IP}&quot;<br>client_addr = &quot;0.0.0.0&quot;<br>bootstrap_expect = 3<br>retry_join = [${CLUSTER_MEMBERS// /,}]<br>ui = true<br>EOF<br><br># Fix retry_join IP format<br>sed -i &#39;s/\([0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\)/&quot;\1&quot;/g&#39; /etc/consul.d/consul.hcl<br><br># systemd service<br>cat &lt;&lt;EOF &gt; /etc/systemd/system/consul.service<br>[Unit]<br>Description=Consul Agent<br>Requires=network-online.target<br>After=network-online.target<br><br>[Service]<br>User=consul<br>ExecStart=/usr/local/bin/consul agent -config-dir=/etc/consul.d/<br>Restart=on-failure<br>EOF<br><br># Start Consul<br>systemctl daemon-reload<br>systemctl enable consul<br>systemctl start consul</pre><p>This script ensures each Consul server has its own <strong>node name</strong> and <strong>private IP</strong>, and the bootstrap_expect=3 setting ensures all three servers join the same cluster.</p><h3>Setting Up the Consul Client</h3><p>For <strong>client nodes</strong> like PgCat, I set up a minimal Consul agent configuration to connect them to the Consul server cluster.</p><pre>#!/bin/bash<br><br>NODE_NAME=&quot;&lt;CLIENT_NODE_NAME&gt;&quot;<br>PRIVATE_IP=&quot;&lt;CLIENT_PRIVATE_IP&gt;&quot;<br>CONSUL_VERSION=&quot;&lt;CONSUL_VERSION&gt;&quot;<br>CLUSTER_MEMBERS=&quot;&lt;SERVER_IP_1&gt; &lt;SERVER_IP_2&gt; &lt;SERVER_IP_3&gt;&quot;<br><br># Install dependencies<br>apt-get update<br>apt-get install -y unzip curl jq<br><br># Download and install Consul<br>curl -O https://releases.hashicorp.com/consul/${CONSUL_VERSION}/consul_${CONSUL_VERSION}_linux_amd64.zip<br>unzip consul_${CONSUL_VERSION}_linux_amd64.zip<br>mv consul /usr/local/bin/<br>chmod +x /usr/local/bin/consul<br><br># Create Consul user and directories<br>useradd --system --home /etc/consul.d --shell /bin/false consul<br>mkdir -p /opt/consul /etc/consul.d<br>chown -R consul:consul /opt/consul /etc/consul.d<br><br># Gossip key (reuse or generate)<br>if [ ! -f /etc/consul.d/gossip.key ]; then<br>  consul keygen &gt; /etc/consul.d/gossip.key<br>fi<br><br># Consul config (client agent)<br>cat &lt;&lt;EOF &gt; /etc/consul.d/consul.hcl<br>datacenter = &quot;dc1&quot;<br>data_dir = &quot;/opt/consul&quot;<br>encrypt = &quot;&lt;GOSSIP_KEY&gt;&quot;<br>node_name = &quot;${NODE_NAME}&quot;<br>server = false<br>bind_addr = &quot;${PRIVATE_IP}&quot;<br>client_addr = &quot;0.0.0.0&quot;<br>retry_join = [${CLUSTER_MEMBERS// /,}]<br>ui = false<br>EOF<br><br># Fix retry_join IP format<br>sed -i &#39;s/\([0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\)/&quot;\1&quot;/g&#39; /etc/consul.d/consul.hcl<br><br># systemd service<br>cat &lt;&lt;EOF &gt; /etc/systemd/system/consul.service<br>[Unit]<br>Description=Consul Agent (Client)<br>Requires=network-online.target<br>After=network-online.target<br><br>[Service]<br>User=consul<br>ExecStart=/usr/local/bin/consul agent -config-dir=/etc/consul.d/<br>Restart=on-failure<br>EOF<br><br># Start the service<br>systemctl daemon-reload<br>systemctl enable consul<br>systemctl start consul</pre><p>The clients can <strong>pull the leader information</strong> and keep their configurations dynamic.</p><p>Consul server cluster automates leader node management<br>Client nodes (like PgCat, K8s etc.) pull <strong>real-time</strong> leader and replica info<br>The entire setup is <strong>highly available</strong> and <strong>flexible</strong></p><p>I hope this helps anyone looking to integrate Consul and PostgreSQL for a resilient architecture. Let’s discuss in the comments if you have any questions!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd0d6c6dc3e6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I Automated PgCat with Consul for Dynamic PostgreSQL Load Balancing?]]></title>
            <link>https://medium.com/@kaanberkozbek/how-i-automated-pgcat-with-consul-for-dynamic-postgresql-load-balancing-399ab0673b8b?source=rss-2d45040d93b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/399ab0673b8b</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[pgcat]]></category>
            <category><![CDATA[consul]]></category>
            <category><![CDATA[high-availability]]></category>
            <dc:creator><![CDATA[Kaan Berk ÖZBEK]]></dc:creator>
            <pubDate>Mon, 02 Jun 2025 00:19:45 GMT</pubDate>
            <atom:updated>2025-06-02T00:22:08.088Z</atom:updated>
            <content:encoded><![CDATA[<p>PgCat is a lightweight and fast connection pooler for PostgreSQL. Thanks to its <strong>read-write splitting</strong> feature, it’s especially useful for balancing workloads between leader and replica nodes. However, in a dynamic cluster — managed by Patroni or similar tools — the leader IP can change frequently. That’s why I wanted to update the PgCat configuration automatically using Consul.</p><p>And that’s where a small but powerful <strong>bash script</strong> comes in!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*TvpR0DMiOfipztf2MhKk8g.png" /><figcaption>Schema — 1</figcaption></figure><h3>The Dynamic Configuration Script</h3><p>The primary purpose of this script is:</p><p>Fetch the current leader node from Consul<br>Detect all running PostgreSQL nodes<br>Rebuild the pgcat.toml file and reload it for PgCat</p><p>Here’s the streamlined flow of the script:</p><pre>#!/bin/bash<br>set -e<br><br>PREVIOUS_LEADER_FILE=&quot;/var/lib/pgcat/last_leader&quot;<br>PREVIOUS_NODES_FILE=&quot;/var/lib/pgcat/last_running_nodes&quot;<br><br># Fetch leader from Consul<br>CURRENT_LEADER=$(curl -s http://127.0.0.1:8500/v1/kv/db/postgres/leader | jq -r &#39;.[0].Value&#39; | base64 -d)<br><br># PostgreSQL node IPs<br>declare -A NODE_IPS=(<br>  [postgre1]=&quot;&lt;IP-1&gt;&quot;<br>  [postgre2]=&quot;&lt;IP-2&gt;&quot;<br>  [postgre3]=&quot;&lt;IP-3&gt;&quot;<br>)<br><br>LEADER_IP=&quot;${NODE_IPS[$CURRENT_LEADER]}&quot;<br>REPLICA_IPS=()<br><br># Check which nodes are up<br>RUNNING_NODES=()<br>for NODE in &quot;${!NODE_IPS[@]}&quot;; do<br>  IP=&quot;${NODE_IPS[$NODE]}&quot;<br>  if timeout 1 bash -c &quot;&lt;/dev/tcp/${IP}/5432&quot; 2&gt;/dev/null; then<br>    RUNNING_NODES+=(&quot;$IP&quot;)<br>    if [[ &quot;$NODE&quot; != &quot;$CURRENT_LEADER&quot; ]]; then<br>      REPLICA_IPS+=(&quot;$IP&quot;)<br>    fi<br>  fi<br>done<br><br># Compare with the previous configuration<br>CURRENT_NODES_STRING=$(IFS=,; echo &quot;${RUNNING_NODES[*]}&quot;)<br>PREVIOUS_LEADER=$(cat &quot;$PREVIOUS_LEADER_FILE&quot; 2&gt;/dev/null || echo &quot;&quot;)<br>PREVIOUS_NODES=$(cat &quot;$PREVIOUS_NODES_FILE&quot; 2&gt;/dev/null || echo &quot;&quot;)<br><br>if [[ &quot;$CURRENT_LEADER&quot; == &quot;$PREVIOUS_LEADER&quot; &amp;&amp; &quot;$CURRENT_NODES_STRING&quot; == &quot;$PREVIOUS_NODES&quot; ]]; then<br>  echo &quot;No changes detected, exiting.&quot;<br>  exit 0<br>fi<br><br># Save updated leader and running nodes<br>echo &quot;$CURRENT_LEADER&quot; &gt; &quot;$PREVIOUS_LEADER_FILE&quot;<br>echo &quot;$CURRENT_NODES_STRING&quot; &gt; &quot;$PREVIOUS_NODES_FILE&quot;<br><br># Rebuild pgcat.toml<br>CONFIG=&quot;/etc/pgcat/pgcat.toml&quot;<br>rm -rf /etc/pgcat<br>mkdir -p /etc/pgcat<br>touch &quot;$CONFIG&quot;<br><br>cat &gt; &quot;$CONFIG&quot; &lt;&lt;EOF<br>[general]<br># Configuration options...<br>EOF<br><br># Generate pools dynamically<br>POOLS=(&quot;&lt;POOL-1&gt;&quot; &quot;&lt;POOL-2&gt;&quot; &quot;&lt;POOL-3&gt;&quot;)<br>USERNAME=&quot;&lt;DB_USER&gt;&quot;<br>PASSWORD=&quot;&lt;DB_PASS&gt;&quot;<br><br>for POOL in &quot;${POOLS[@]}&quot;; do<br>  cat &gt;&gt; &quot;$CONFIG&quot; &lt;&lt;EOF<br><br>[pools.${POOL}]<br>pool_mode = &quot;transaction&quot;<br># Other pool settings...<br><br>[pools.${POOL}.users.0]<br>username = &quot;${USERNAME}&quot;<br>password = &quot;${PASSWORD}&quot;<br>pool_size = 20<br><br>[pools.${POOL}.shards.0]<br>servers = [<br>    [&quot;${LEADER_IP}&quot;, 5432, &quot;primary&quot;],<br>EOF<br><br>  for IP in &quot;${REPLICA_IPS[@]}&quot;; do<br>    echo &quot;    [\&quot;${IP}\&quot;, 5432, \&quot;replica\&quot;],&quot; &gt;&gt; &quot;$CONFIG&quot;<br>  done<br><br>  sed -i &#39;$ s/,$//&#39; &quot;$CONFIG&quot;<br><br>  cat &gt;&gt; &quot;$CONFIG&quot; &lt;&lt;EOF<br>]<br>database = &quot;${POOL}&quot;<br>EOF<br>done<br><br># Reload PgCat with SIGHUP<br>docker kill -s HUP pgcat<br></pre><h3>Automating with systemd Services and Timers</h3><p>Instead of running the bash script manually, I automated it using a systemd service and timer:</p><pre># /etc/systemd/system/update-pgcat.service<br>[Unit]<br>Description=Update PgCat backend configuration from Consul<br><br>[Service]<br>ExecStart=/usr/local/bin/update-pgcat-config.sh<br>Type=oneshot</pre><pre># /etc/systemd/system/update-pgcat.timer<br>[Unit]<br>Description=Periodic PgCat config update<br><br>[Timer]<br>OnBootSec=10s<br>OnUnitActiveSec=15s<br>Unit=update-pgcat.service<br><br>[Install]<br>WantedBy=timers.target</pre><p>This ensures that the system automatically detects leader changes every 15 seconds and updates the PgCat configuration accordingly. So whenever there’s a new leader or a replica comes online, PgCat instantly reflects these changes!</p><h3>Results and My Takeaways</h3><p>By following this approach:</p><p>I seamlessly integrated Consul + PgCat + Docker for high availability<br>PgCat’s config always matches the current cluster topology<br>I kept the solution minimal — just bash and systemd did the job!</p><p>Of course, I’ve replaced sensitive IP addresses and credentials with in this article. Make sure to update these in your environment.</p><p>If you’re interested in PostgreSQL cluster management or want to discuss Consul-PgCat integration, let’s connect in the comments!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/112/1*muOg55PfukjvdqYi2jKPag.png" /><figcaption>Schema — 2</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=399ab0673b8b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Top APM Tools: Real-World Comparison of Pros, Cons, Cost, Performance, and Use Cases]]></title>
            <link>https://medium.com/@kaanberkozbek/top-apm-tools-real-world-comparison-of-pros-cons-cost-performance-and-use-cases-b7dc77f97102?source=rss-2d45040d93b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/b7dc77f97102</guid>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[dynatrace]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[splunk]]></category>
            <category><![CDATA[apm]]></category>
            <dc:creator><![CDATA[Kaan Berk ÖZBEK]]></dc:creator>
            <pubDate>Tue, 13 May 2025 15:26:19 GMT</pubDate>
            <atom:updated>2025-05-14T20:05:47.190Z</atom:updated>
            <content:encoded><![CDATA[<p>Application Performance Monitoring (APM) tools have become indispensable in today’s complex software ecosystems. Whether you’re managing microservices in the cloud or legacy systems on-premises, visibility into system behavior is crucial. In my experience as a DevOps engineer, APM platforms serve as the foundation for diagnosing performance issues, correlating telemetry, and maintaining service reliability. This article presents a real-world, hands-on comparison of top APM tools in 2025 — covering their strengths, weaknesses, pricing structure, scalability, and where they best fit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/553/1*elSTMuak_9WOZ0ai_wPUNw.png" /><figcaption>Pic 1</figcaption></figure><h3>1. Dynatrace</h3><p>Dynatrace is an enterprise-grade observability platform that stands out with its powerful AI engine, Davis, and fully automated instrumentation via OneAgent. It seamlessly discovers applications and dependencies with minimal configuration, offering deep insights across the entire stack.</p><h4>Pros:</h4><ul><li><strong>Automated setup</strong>: OneAgent discovers and monitors applications automatically.</li><li><strong>AI-powered root cause analysis</strong>: Davis correlates telemetry and pinpoints issues with high precision.</li><li><strong>Scalability</strong>: Ideal for large-scale deployments; widely used in Fortune 500 companies.</li><li><strong>Comprehensive tech support</strong>: From modern stacks (Kubernetes, containers) to legacy (mainframes).</li><li><strong>All-in-one observability</strong>: APM, infrastructure, logs, RUM, and security in one platform.</li></ul><h4>Cons:</h4><ul><li><strong>Premium cost model</strong>: Pricing is usage-based and tailored for enterprise-scale setups, which can be high for large environments.</li><li><strong>Complex UI for newcomers</strong>: Dense with data and features; onboarding takes time.</li><li><strong>Agent resource usage</strong>: May require tuning in high-performance environments.</li><li><strong>Limited on-prem control</strong>: Fully self-managed installations are not fully isolated.</li></ul><h4>Pricing Model:</h4><p>Dynatrace uses a host-based and feature-tiered pricing system. Units are billed based on memory size and enabled features (like APM, logs, or RUM). Enterprises typically negotiate custom contracts. A short free trial is available, but there’s no ongoing free tier.</p><h4>Performance &amp; Scalability:</h4><p>Designed to handle high volumes of unsampled data with real-time analysis. Capable of monitoring thousands of services without sacrificing data granularity.</p><h4>Best Use Cases:</h4><p>Best suited for large enterprises with hybrid environments, Kubernetes clusters, and complex stacks. Also ideal for lean teams looking for automated diagnostics.</p><h4>Setup:</h4><p>Deployment is straightforward via shell script or Helm (for Kubernetes). Agents auto-inject and begin monitoring immediately. Offers seamless integration with AWS, Azure, GCP, OpenShift, and more.</p><h3>2. Datadog</h3><p>Datadog has become a favorite among cloud-native teams due to its ease of use and broad feature set. It started with infrastructure monitoring but now covers APM, logs, RUM, synthetics, security, and more — all accessible through a unified web interface.</p><h3>Pros:</h3><ul><li><strong>Comprehensive observability</strong>: Metrics, traces, logs, security, and more in a single platform.</li><li><strong>Easy setup</strong>: SaaS model with simple agent deployment.</li><li><strong>Extensive integrations</strong>: Supports over 500 technologies with minimal manual work.</li><li><strong>Custom dashboards</strong>: Rich templates and visualizations out-of-the-box.</li><li><strong>Scales with ease</strong>: Cloud architecture handles large telemetry volumes seamlessly.</li></ul><h4>Cons:</h4><ul><li><strong>Expensive at scale</strong>: Costs add up with each feature and host. Must monitor usage carefully.</li><li><strong>Overwhelming UI for smaller teams</strong>: Feature-rich UI can be complex without proper training.</li><li><strong>Data retention trade-offs</strong>: Extended granularity or retention comes at an additional cost.</li><li><strong>Primarily cloud-based</strong>: Limited support for fully air-gapped, on-prem-only setups.</li></ul><h4>Pricing Model:</h4><p>Datadog follows a modular, usage-based model where each product (APM, logs, etc.) adds to the cost. The more features and hosts you monitor, the higher the bill. While the pricing is flexible, large environments must budget and monitor usage carefully. A free trial and limited free tier are available.</p><h4>Performance &amp; Scalability:</h4><p>Designed for modern cloud workloads, Datadog handles high-throughput telemetry and real-time alerting efficiently. Agent overhead is minimal, and the UI supports dynamic queries across large datasets.</p><h4>Best Use Cases:</h4><p>Great for DevOps/SRE teams in AWS, Azure, or GCP environments. Suited for teams using Kubernetes, serverless, or distributed systems who want a managed solution. Less ideal for strict on-prem environments or early-stage startups with budget constraints.</p><h4>Setup:</h4><p>Setup is fast via shell script or Helm. The Datadog agent supports automatic instrumentation for many services and frameworks. Integrates well with CI/CD and major cloud platforms.</p><h3>3. New Relic</h3><p>New Relic is one of the pioneers in the APM space and has evolved into a full-stack observability platform. I’ve used it on and off for over a decade, and in 2025 it continues to deliver reliable insights across applications, infrastructure, and user experience.</p><h4>Pros:</h4><ul><li><strong>Unified Observability</strong>: New Relic One integrates APM, logs, infrastructure, synthetics, and browser monitoring into a single platform, with intuitive dashboards and minimal setup friction.</li><li><strong>Deep APM Insights</strong>: Offers granular, code-level instrumentation out-of-the-box. Supports many languages and provides details on function timings, DB calls, and external dependencies.</li><li><strong>Developer-Friendly Pricing</strong>: Usage-based with a generous free tier, offering a lot of value for small teams. Pricing is based on data ingestion and user type, not per-host, which makes it attractive for high-host-count environments.</li><li><strong>Strong for Both Dev and Ops</strong>: Equally useful for debugging slow code and monitoring overall system health. RUM and synthetics tie frontend and backend performance together.</li><li><strong>OpenTelemetry Support</strong>: Accepts OpenTelemetry traces, enabling integration with open standards.</li></ul><h4>Cons:</h4><ul><li><strong>Complex Cost Scaling</strong>: While predictable at small scale, pricing can escalate with heavy data ingest or multiple full-access users. Teams must monitor usage to avoid surprise costs.</li><li><strong>UI Learning Curve</strong>: After consolidations and UI redesigns, finding key settings or data may take some getting used to.</li><li><strong>Agent Overhead</strong>: Detailed instrumentation can lead to non-trivial overhead on JVM or .NET apps, especially under high load.</li><li><strong>Less Popular Among Startups</strong>: Newer teams sometimes view it as a legacy tool, despite its modern capabilities.</li></ul><h4>Pricing Model:</h4><p>New Relic’s pricing is usage-based, focusing on data volume and user access levels. It offers a generous free tier, and beyond that, users pay based on how much data they ingest and the number/type of users (Full, Core, Basic). Host count doesn’t affect billing directly, making it predictable for large infrastructures with low-to-moderate traffic.</p><h4>Performance &amp; Scalability:</h4><p>The SaaS backend scales well for high volumes of telemetry, with smart sampling and retention strategies. APM data remains granular and queryable for meaningful periods, and the asynchronous nature of its agents keeps app impact low.</p><h4>Best Use Cases:</h4><p>Ideal for startups leveraging the free tier, mid-size teams needing end-to-end observability without per-host costs, or large orgs that want rich APM tied to user experience. Particularly strong for polyglot environments and application-focused teams.</p><h4>Setup:</h4><p>Install language-specific agents, configure a license key, and deploy. For Kubernetes, use Helm or DaemonSets. The platform also includes quickstart bundles for common stacks (NGINX, MongoDB, etc.), which speed up initial dashboard creation.</p><h3>4. Elastic APM</h3><p>Elastic APM is the open-source observability tool built into the Elastic Stack. For teams already running Elasticsearch for logs or metrics, Elastic APM is a natural and cost-effective extension for tracing application performance.</p><h4>Pros:</h4><ul><li><strong>Free &amp; Open Source</strong>: You can self-host Elastic APM at no licensing cost, ideal for cost-sensitive projects or on-prem deployments.</li><li><strong>Seamless ELK Integration</strong>: APM, logs, and metrics live in one place — Kibana — enabling fast correlation and full visibility across your stack.</li><li><strong>Flexible Storage &amp; Retention</strong>: Full control over how much data to store, how long, and what to index. Perfect for custom retention strategies.</li><li><strong>Supports OpenTelemetry &amp; Multiple Languages</strong>: Can receive OTel data and has agents for Java, Python, Node, Go, .NET, and more.</li></ul><h4>Cons:</h4><ul><li><strong>Ops Overhead</strong>: Requires managing Elasticsearch, Kibana, and APM Server, which demands time, tuning, and operational expertise.</li><li><strong>Scaling Can Be Hard</strong>: Without proper indexing or sampling, high trace volume can stress your Elasticsearch cluster.</li><li><strong>Fewer Enterprise Features</strong>: Lacks built-in AI diagnostics or native business transaction views found in premium tools.</li><li><strong>Feature Licensing</strong>: Some features like anomaly detection or fine-grained RBAC require a paid Elastic license or Elastic Cloud subscription.</li></ul><h4>Pricing Model:</h4><p>Self-hosted Elastic APM is free. If you choose Elastic Cloud, pricing is based on resource usage (e.g., nodes, data tiers). Enterprise features (like ML or security) are gated behind tiered licenses. Overall, you trade software costs for infrastructure and ops effort.</p><h4>Performance &amp; Scalability:</h4><p>Performance depends on cluster tuning. Mid-size workloads are handled well, but for high-volume environments, horizontal scaling and trace sampling are needed. With proper configuration, it can ingest thousands of events per second.</p><h4>Best Use Cases:</h4><p>Ideal for teams already using the Elastic Stack, or those who need an on-prem, customizable APM solution. Also great for orgs that prioritize data sovereignty or want to avoid vendor lock-in.</p><h4>Setup:</h4><p>Install Elasticsearch, Kibana, and APM Server. Add agents to applications, configure endpoints, and start viewing data in Kibana’s APM app. Kubernetes Helm charts and Docker Compose setups are available for fast deployment.</p><h3>5. Sentry</h3><p>Sentry is a developer-centric monitoring tool that started with error tracking but now offers performance monitoring as well. While not a full-stack APM, it excels in helping developers debug what’s happening inside their code.</p><h4>Pros:</h4><ul><li><strong>Top-Tier Error Monitoring</strong>: Automatically captures stack traces, groups issues, and integrates with dev tools like Slack, Jira, and GitHub.</li><li><strong>Performance Insights for Code</strong>: Traces show slow spans, inefficient DB queries, and N+1 issues — perfect for API and frontend debugging.</li><li><strong>Quick Setup</strong>: One of the easiest tools to integrate, with SDKs for most major backends and frontends.</li><li><strong>User Context</strong>: Connects performance and errors with individual user sessions, enhancing debugging relevance.</li><li><strong>Free Tier &amp; Affordable Pricing</strong>: Great for small apps or early-stage projects. Self-hosting is also available.</li></ul><h4>Cons:</h4><ul><li><strong>Not Infrastructure-Focused</strong>: Doesn’t monitor CPU, memory, or network stats — it’s not a replacement for infra tools.</li><li><strong>Limited Distributed Tracing</strong>: Cross-service tracing exists, but it’s less seamless compared to full APM tools.</li><li><strong>Self-Hosting at Scale is Complex</strong>: Running Sentry yourself involves Kafka, Postgres, ClickHouse, etc., which requires expertise.</li><li><strong>Mainly Web &amp; API Focus</strong>: Mobile support exists for crash tracking, but performance monitoring is more tuned to web stacks.</li></ul><h4>Pricing Model:</h4><p>Sentry uses an event-based model. The free plan includes a limited number of error and transaction events. Paid tiers increase event capacity, data retention, and team features. Pricing remains lower than most enterprise APMs, making it very startup-friendly.</p><h4>Performance &amp; Scalability:</h4><p>Sentry Cloud handles billions of events per month with minimal latency. The SDKs send data asynchronously and support sampling to control overhead. The UI is responsive and optimized for common developer workflows.</p><h4>Best Use Cases:</h4><p>Perfect for dev teams needing insight into application-level performance and exceptions. Great for CI/CD pipelines, frontend performance, and post-deploy debugging. Pairs well with infra-focused tools like Prometheus or Datadog.</p><h4>Setup:</h4><p>Register your app with a DSN and add the SDK. For backend: sentry_sdk.init(...); for frontend: Sentry.init(...). That’s it. Alerts, dashboards, and source maps can be configured through the UI. Self-hosting available via Docker Compose.</p><h4>6.AppDynamics</h4><p>AppDynamics (AppD), part of Cisco since 2017, is a veteran enterprise APM solution known for its deep correlation between application performance and business metrics. In several large-scale Java/.NET deployments I’ve managed, AppD was invaluable for diagnosing issues at the transaction level while also providing insights aligned with KPIs like “Checkout Success Rate” or “Funds Transfer Latency.”</p><h4>Pros:</h4><ul><li><strong>Business Transaction Focus</strong>: Automatically detects logical flows (like “Login” or “Checkout”) across services, providing business-context to performance data.</li><li><strong>Granular Code-Level Analysis</strong>: Deep visibility into methods, DB queries, and error traces without code changes.</li><li><strong>Enterprise-Ready with On-Prem Option</strong>: Ideal for regulated or air-gapped environments, with full on-prem deployment possible.</li><li><strong>Smart Baseline Alerting</strong>: Learns normal behavior patterns to reduce false positives.</li><li><strong>Extensive Platform Support</strong>: Works across legacy and modern stacks, supports SAP, custom protocols, and integrates with cloud environments.</li></ul><h4>Cons:</h4><ul><li><strong>Complex Deployment</strong>: Requires careful planning, especially when self-hosted; agents and Controller must be tuned.</li><li><strong>Dated UI</strong>: Usable, but feels older compared to newer APM UIs.</li><li><strong>High Licensing Costs</strong>: Priced per agent/node, with modules sold separately. No free tier — built for enterprise budgets.</li><li><strong>Overhead and Resource Demand</strong>: Can introduce 5–10% CPU load unless optimized.</li><li><strong>Roadmap Uncertainty</strong>: Cisco’s acquisition of Splunk introduces potential product convergence questions.</li></ul><h4>Pricing Model:</h4><p>Sold via enterprise contracts, usually per agent/unit. Expect a premium price. Frequently deployed only for mission-critical apps due to cost. No free usage, only trials.</p><h4>Performance &amp; Scalability:</h4><p>Supports massive environments with proper resource allocation. AppD’s sampling strategy offers detailed snapshots without overwhelming storage. Real-time insights at 1-minute granularity for metrics, finer for traces.</p><h4>Best Use Cases:</h4><p>Enterprises needing detailed performance tied to business outcomes — especially in finance, telecom, or retail. Ideal when full-stack, on-prem observability is required, or where business KPIs must be monitored alongside technical metrics.</p><h4>Setup:</h4><p>Install Controller (on-prem or cloud), deploy language-specific agents, define application tiers. Requires tuning, especially for JVMs or .NET apps, but yields powerful diagnostics once running.</p><h3>7. Riverbed (Alluvio Aternity)</h3><p>Alluvio Aternity by Riverbed blends APM with Digital Experience Monitoring (DEM), giving unique visibility into both backend services and the end-user experience on devices like desktops and laptops. While niche, it excels in enterprise IT environments focused on user productivity and client performance.</p><h4>Pros:</h4><ul><li><strong>End-to-End Monitoring</strong>: Connects device experience with backend performance — rare in the APM world.</li><li><strong>100% Transaction Capture</strong>: Unsampled, high-granularity tracing ensures no blind spots.</li><li><strong>Excellent Device Monitoring</strong>: Provides PC performance metrics and app responsiveness for every user.</li><li><strong>Network-Aware</strong>: Integrates with Riverbed network tools for full path visibility from client to server.</li><li><strong>Well-Rated by Users</strong>: Despite a small footprint, customer satisfaction is high.</li></ul><h4>Cons:</h4><ul><li><strong>Limited Ecosystem</strong>: Small market share means fewer third-party resources or experienced users.</li><li><strong>Heavy Deployment Footprint</strong>: Requires agents on devices and backend, plus several components to deploy.</li><li><strong>UI Overload</strong>: Rich detail can be overwhelming without tuning.</li><li><strong>Not Cloud-Native First</strong>: Best suited for traditional apps and desktop software, not Kubernetes-heavy stacks.</li><li><strong>Enterprise Pricing Model</strong>: Designed for large orgs, likely expensive and license-based per device or agent.</li></ul><h4>Pricing Model:</h4><p>Enterprise-tier pricing by agent/device. No public free tier. Often bundled with other Riverbed solutions.</p><h4>Performance &amp; Scalability:</h4><p>Scales to tens of thousands of endpoints with appropriate backend infrastructure. Designed for high-resolution telemetry and real-time feedback, but requires well-planned architecture to avoid overload.</p><h4>Best Use Cases:</h4><p>Perfect for IT teams needing visibility into internal application performance from the user’s perspective. Strong fit for industries like finance or healthcare where desktop experience matters as much as backend health.</p><h4>Setup:</h4><p>Deploy Aternity server or use SaaS. Install agents on user devices and app servers. Configure transaction definitions and collect telemetry. Riverbed offers professional services for large rollouts.</p><h3>8. Splunk Observability Cloud</h3><p>Born from acquisitions like SignalFx and Omnition, Splunk Observability Cloud is a cloud-native powerhouse built for real-time, unsampled, full-fidelity observability. It shines in high-scale environments where missing a single anomaly isn’t acceptable.</p><h4>Pros:</h4><ul><li><strong>100% Trace Retention</strong>: Captures every span, every time — crucial for debugging rare, complex issues.</li><li><strong>Real-Time Analytics</strong>: Dashboards refresh every second, ideal for incident response.</li><li><strong>Strong AI/ML Features</strong>: Smart alerting and anomaly detection with predictive insights.</li><li><strong>OpenTelemetry-Native</strong>: Fully embraces open standards — highly flexible and future-proof.</li><li><strong>Integrated Observability Suite</strong>: Combines infra, APM, logs, RUM, and synthetics in one place</li></ul><h4>Cons:</h4><ul><li><strong>Premium Pricing</strong>: Cost grows with data ingestion volume; no free tier. Best suited for orgs with serious budgets.</li><li><strong>Steep Learning Curve</strong>: Advanced features (like custom detectors or SignalFlow queries) require training.</li><li><strong>SaaS-Only</strong>: Not available for full on-prem deployment.</li><li><strong>Split UX with Splunk Enterprise</strong>: Some features live outside the classic Splunk UI.</li></ul><h4>Pricing Model:</h4><p>Fully usage-based: traces, spans, metrics, logs — everything is metered. Requires careful planning to avoid runaway bills. Designed for enterprise customers and negotiated contracts.</p><h4>Performance &amp; Scalability:</h4><p>Best-in-class ingestion and query latency. Handles high cardinality metrics and massive span volumes with ease. 1-second resolution across the stack.</p><h4>Best Use Cases:</h4><p>Best suited for high-frequency, distributed environments (e.g., fintech, SaaS, telco) needing real-time, full-fidelity observability. Also logical for existing Splunk log users seeking deeper app visibility.</p><h4>Setup:</h4><p>Deploy Splunk Otel Collector (Helm, VM agent, or sidecar). Configure exporters and receivers. Supports Otel SDKs and integration with custom pipelines. Requires initial planning but rewards with flexibility and scale.</p><h3>9. Instana (IBM Observability by Instana)</h3><p>Instana, now backed by IBM, is a modern APM solution that emphasizes automation, ease of setup, and high-resolution monitoring. It’s often positioned as the “just works” alternative to Dynatrace — still full-featured, but with faster onboarding and transparent pricing.</p><h4>Pros:</h4><ul><li><strong>Instant Deployment</strong>: Auto-discovers services, injects tracing, and builds dashboards within minutes.</li><li><strong>1s Metric Resolution</strong>: Extremely granular and low-latency insights.</li><li><strong>AI-Driven Root Cause Analysis</strong>: Automatically correlates issues and reduces noise.</li><li><strong>Modern UI &amp; Architecture</strong>: Built for containers, Kubernetes, and microservices.</li><li><strong>Transparent Licensing</strong>: Generally priced per host/unit, with all features included.</li></ul><h4>Cons:</h4><ul><li><strong>Still Enterprise-Centric</strong>: Lower cost than some, but still not budget-tier.</li><li><strong>Smaller Community</strong>: Fewer third-party integrations or blog content than larger rivals.</li><li><strong>IBM Alignment Risk</strong>: Roadmap could shift depending on IBM strategy.</li><li><strong>Limited Log Management</strong>: Not a full log solution (yet); needs external log tooling.</li></ul><h4>Pricing Model:</h4><p>Host-based with predictable tiers. Pricing includes metrics, traces, and core analytics. Competitive in enterprise space, with fewer add-ons than others. Free trial available.</p><h4>Performance &amp; Scalability:</h4><p>Real-time with high data resolution. Agents auto-tune and minimize overhead. Scales horizontally for massive workloads. Offers both SaaS and self-hosted deployment.</p><h4>Best Use Cases:</h4><p>Perfect for Kubernetes-native teams, CI/CD-heavy environments, or mid-sized companies wanting powerful APM with less setup. Also great for enterprises aligned with IBM or looking to reduce APM complexity.</p><h4>Setup:</h4><p>Install a single agent per host (or DaemonSet in K8s). Agents auto-instrument most languages and services. For self-hosted, deploy containerized backend. UI quickly fills with topology and service maps.</p><h3>10. Prometheus &amp; Grafana (Open-Source DIY APM Stack)</h3><p>Prometheus and Grafana form the backbone of many open-source observability stacks. While not a full APM suite out-of-the-box, this duo — when combined with projects like Jaeger, Tempo, and Loki — can be assembled into a comprehensive monitoring solution. I’ve used Prometheus+Grafana extensively for Kubernetes infrastructure and application metrics, and with some effort, it can handle a lot of what commercial APMs do, albeit in a more DIY fashion.</p><p><strong>Pros:</strong></p><ul><li><strong>Open-source and cost-effective:</strong> No license fees. You control everything.</li><li><strong>Highly customizable:</strong> PromQL is powerful, and Grafana’s dashboarding is second to none.</li><li><strong>Ecosystem-rich:</strong> Hundreds of exporters, support for tracing/logging through add-ons.</li><li><strong>Kubernetes-native:</strong> First-class integration with K8s and service discovery.</li><li><strong>Scalable via tools like Thanos/Cortex:</strong> Supports large deployments if properly designed.</li></ul><p><strong>Cons:</strong></p><ul><li><strong>Not plug-and-play:</strong> Requires engineering time to stitch components together.</li><li><strong>No unified APM view:</strong> Traces/metrics/logs often live in separate tools unless tightly integrated.</li><li><strong>Maintenance overhead:</strong> You’ll need to manage scaling, persistence, and upgrades.</li><li><strong>Basic alerting:</strong> No built-in anomaly detection or smart thresholds.</li><li><strong>Steep learning curve:</strong> PromQL, alerting rules, and instrumentation all demand time and expertise.</li></ul><p><strong>Pricing model:</strong> Free software. You pay for infrastructure and engineering effort. Managed services like Grafana Cloud offer convenience at a low cost. Still, one of the most affordable options overall — ideal for cost-conscious teams with DevOps skills.</p><p><strong>Best for:</strong> Teams running Kubernetes or modern cloud-native stacks who want observability flexibility without vendor lock-in. If you’re comfortable with a bit of DIY and want full control, this is the stack to start with.</p><h3>11. SigNoz (Open-Source APM Alternative)</h3><p>SigNoz is part of a new generation of open-source APMs that aim to deliver a Datadog-like experience without the commercial overhead. Built on top of OpenTelemetry and ClickHouse, it offers traces, metrics, and logs in a single pane — self-hosted and open-source. I’ve experimented with SigNoz and closely followed its community; it’s impressive how much it offers out-of-the-box.</p><p><strong>Pros:</strong></p><ul><li><strong>Truly open-source (Apache 2.0):</strong> No lock-in. Built for self-hosting.</li><li><strong>Unified observability:</strong> Traces, metrics, and logs are all in one UI.</li><li><strong>Otel-native:</strong> Works seamlessly with OpenTelemetry SDKs and collectors.</li><li><strong>Developer-friendly setup:</strong> Docker and Helm support, clean UI, and prebuilt dashboards.</li><li><strong>Cost-effective at mid-scale:</strong> Efficient ClickHouse backend handles decent workloads affordably.</li></ul><p><strong>Cons:</strong></p><ul><li><strong>Still maturing:</strong> Not yet battle-tested at massive enterprise scale.</li><li><strong>Self-hosting burden:</strong> Needs DevOps effort to run, update, and monitor.</li><li><strong>Limited enterprise features:</strong> Things like RBAC, SSO, deep integrations are still growing.</li><li><strong>UI not as polished as big players:</strong> Lacks the insight depth of AppDynamics or Dynatrace (yet).</li><li><strong>Log management scale limits:</strong> Full logging can be resource-heavy unless tuned.</li></ul><p><strong>Pricing model:</strong> Free if self-hosted. Paid managed version available. Primary costs are infrastructure and maintenance. One of the best value-to-feature ratios among open-source APM tools today.</p><p><strong>Best for:</strong> Startups, small-to-mid-sized teams, and privacy-conscious orgs looking for an integrated APM platform without commercial licenses. Great for OpenTelemetry adopters who want a ready-made backend with a solid UI.</p><h3>Comparison Table</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZsXkG7uTxt1WECZLEGkYeA.png" /><figcaption>Table of 11 APM Tools</figcaption></figure><h3>Final Recommendations</h3><p>After working across startups, scale-ups, and enterprises, I’ve come to realize there’s no universal “best” APM tool — only the best fit for <strong>your</strong> team’s size, budget, tech stack, and appetite for complexity.</p><p>Here’s my decision matrix based on real-world experience:</p><h4>Small Team / Startup (Low Budget, Fast Iteration)</h4><p>Use open-source tools and free tiers:</p><ul><li><strong>[Prometheus + Grafana]</strong> for infra metrics</li><li><strong>[Sentry]</strong> for exception &amp; performance insights</li><li><strong>[SigNoz]</strong> if you want full-stack APM without vendor lock-in (host it yourself)</li></ul><blockquote><strong><em>Avoid</em></strong><em> expensive suites early on. Instrument with </em><strong><em>OpenTelemetry</em></strong><em> so you can switch later.</em></blockquote><p><em>Use case</em>: A SaaS startup with 5 microservices can run Prometheus/Grafana for infra and use Sentry for app errors — all without a monthly bill.</p><h4>Mid-Sized Company (Cloud-Native, Some Budget, DevOps-Savvy)</h4><p>Balance flexibility with ease of use:</p><ul><li><strong>[Datadog]</strong>: Unified SaaS observability, great integrations (but watch cost sprawl)</li><li><strong>[New Relic]</strong>: Cost-effective for high-host, low-ingest environments (user-based pricing.</li><li><strong>[SigNoz]</strong>: Sweet spot for open-source + nice UI</li><li><strong>[Instana]</strong>: Quick APM setup, IBM-backed, good pricing model</li></ul><blockquote><strong><em>SaaS = fewer ops headaches</em></strong><em>, OSS = more control but more work.</em></blockquote><p><em>Use case</em>: A growing platform with 50+ microservices on AWS — <strong>Datadog</strong> or <strong>Instana</strong> for convenience, or <strong>SigNoz</strong> if budget-conscious.</p><h4>Large Enterprise (Mission-Critical, Compliance-Heavy)</h4><p>Look for proven, scalable tools with on-prem support:</p><ul><li><strong>[Dynatrace]</strong>: AI-powered, scalable, great for hybrid/cloud-native at scale</li><li><strong>[AppDynamics]</strong>: Strong for legacy stacks &amp; business transaction monitoring</li><li><strong>[Instana]</strong>: Modern, enterprise-grade, IBM-aligned</li><li><strong>[Splunk Observability]</strong>: Full-fidelity, ideal if already using Splunk logs</li><li><strong>[Riverbed Aternity]</strong>: For endpoint + user experience monitoring</li></ul><blockquote>Legacy-heavy?<em> Go </em><strong><em>AppDynamics</em></strong><em>.<br> </em>Cloud-first?<em> Choose </em><strong><em>Dynatrace</em></strong><em> or </em><strong><em>Splunk.</em></strong><em><br> </em>IBM client or OpenShift user?<em> Use </em><strong><em>Instana</em></strong><em>.</em></blockquote><p><em>Use case</em>: A Fortune 500 bank running legacy + cloud-native — <strong>Dynatrace Managed</strong> with <strong>ActiveGate</strong> for mainframe, or <strong>AppD + Splunk</strong> if deep Cisco/Splunk ecosystem.</p><h4>Cloud-Native Startups (With Funding, Rapid Growth)</h4><p>Time-to-value trumps everything:</p><ul><li><strong>[Dynatrace SaaS]</strong>: Great automation, fast incident resolution</li><li><strong>[Instana SaaS]</strong>: Developer-friendly, low-maintenance, great for K8s</li><li><strong>[Datadog]</strong>: Broad and fast, but needs usage discipline</li></ul><blockquote><em>Invest early in observability if you’re scaling fast. Prevents burnout and blind spots.</em></blockquote><p><em>Use case</em>: A Series B startup with 200 microservices — <strong>Instana</strong> for visibility with minimal setup overhead.</p><h4>Teams Focused on End-User Device Performance</h4><p>Prioritize Digital Experience Monitoring:</p><ul><li><strong>[Riverbed Aternity]</strong>: Best for EUEM (employee endpoints, thick clients)</li><li>Pair it with <strong>AppD</strong>, <strong>New Relic</strong>, or <strong>Dynatrace</strong> for backend APM</li></ul><p><em>Use case</em>: Healthcare org with clinics using EMR software — <strong>Aternity</strong> for client device health, <strong>AppD</strong> for backend performance.</p><h4>Hybrid Strategy</h4><p>Consider mixing:</p><ul><li><strong>Sentry + Prometheus</strong></li><li><strong>Datadog + Lightstep</strong></li><li><strong>Grafana Cloud + AWS X-Ray</strong></li></ul><blockquote><em>You don’t need to marry one tool. Build your own observability stack if needed — but beware “dashboard sprawl.”</em></blockquote><h3>Practical Matrix to Guide Your Choice</h3><ul><li>If <strong>data must not leave our data center</strong> (compliance), immediately SaaS services are out — focus on Dynatrace Managed, AppDynamics, Instana on-prem, or open-source.</li><li>If <strong>we lack dedicated ops for the tool</strong>, lean SaaS or very easy solutions (Datadog, New Relic, Instana SaaS).</li><li>If <strong>cost is the #1 factor</strong>, open source (Prom/Grafana, SigNoz) or New Relic’s free tier to start, and plan to upgrade as needed.</li><li>If <strong>breadth of monitoring</strong> (infrastructure + APM + logs + synthetics in one) is desired and cost is secondary, Datadog or Splunk Observability or Dynatrace will make the team’s life easiest due to one integrated platform.</li><li>If <strong>we’re a Microsoft/.NET shop</strong>, historically New Relic and AppDynamics have strong .NET support; Dynatrace does as well now. But also Azure’s native Application Insights might suffice and be cheaper (I’ve seen some mid-size Windows-heavy teams use Azure Monitor for APM to avoid third-party). So consider cloud-vendor native options in homogeneous environments (AWS X-Ray, etc.) though they often aren’t as full-featured.</li></ul><h3>Final Thoughts</h3><p>The best APM tool isn’t the most expensive or most feature-rich. It’s the one your <strong>team will actually use</strong>. If your developers ignore the dashboard, alerts will rot. So pick a tool that fits your <strong>culture</strong>, not just your stack.</p><p><strong>Start small, stay open-source if you can, and grow into enterprise tools when the cost is justified. </strong>Evaluate tools via <strong>PoCs</strong>. If it’s hard to deploy on two services, it’ll be worse at 200.</p><p>In 2025, the APM ecosystem is rich, mature, and flexible. Whether you need full-fidelity tracing, tight control, or an instant SaaS solution, there’s a path forward.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/112/1*sIPQj-fYQ2Ksbg8BsTesAw.png" /><figcaption>Thanks for reading.</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b7dc77f97102" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[High Avalaible Monolith Posgresql Cluster with Patroni&Consul&Haproxy]]></title>
            <link>https://blog.stackademic.com/high-avalaible-monolith-posgresql-cluster-with-patroni-consul-haproxy-42e37f27e383?source=rss-2d45040d93b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/42e37f27e383</guid>
            <category><![CDATA[pgbouncer]]></category>
            <category><![CDATA[haproxy]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[patroni]]></category>
            <category><![CDATA[postgresql]]></category>
            <dc:creator><![CDATA[Kaan Berk ÖZBEK]]></dc:creator>
            <pubDate>Mon, 14 Apr 2025 22:26:35 GMT</pubDate>
            <atom:updated>2025-04-21T22:42:04.314Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>High Avalaible Monolith Postgresql Cluster with Patroni&amp;Consul&amp;Haproxy</strong></h3><p><strong>Architecture Overview</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M-GyxNNL6t7IT_enXgXbeQ.png" /><figcaption>Topology</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/895/1*Gh4V-Mt39JQ8xEtsUbXiFw.png" /><figcaption>Overview</figcaption></figure><ul><li>The <strong>database cluster</strong> runs on db1–db3 with Patroni and Consul, supporting automatic failover.</li><li>The <strong>routing and access layer</strong> is handled by HAProxy + PgBouncer on proxy1 and proxy2.</li><li><strong>Single point of access</strong> is provided by the virtual IP 10.0.0.30, always directed to the active proxy.</li><li>The <strong>pgAdmin interface</strong> is hosted only on proxy1.</li></ul><p>The main goal of this setup is to eliminate single points of failure and ensure the PostgreSQL database service is continuously accessible. Each component in the architecture has a backup, and the system remains operational even if any part fails.</p><p>The following components and roles are used:</p><ul><li><strong>PostgreSQL + Patroni (3 nodes)</strong>: These are the database nodes. Patroni manages the PostgreSQL cluster, determines the leader (primary) node, and handles automatic failover in case of failure. Patroni uses Consul as a distributed configuration store (DCS).</li><li><strong>Consul (3 nodes, co-located with PostgreSQL nodes)</strong>: Acts as the distributed key-value store required by Patroni for high availability. It maintains cluster node status and the leadership lock. A 3-node Consul cluster provides consistency using quorum-based decision-making (odd number rule).</li><li><strong>HAProxy (2 nodes)</strong>: Serves as the routing and load-balancing layer. All application requests pass through HAProxy. It tracks the current primary (writable) node in the cluster and always forwards clients to the active leader. Optionally, it can distribute read loads across replica nodes.</li><li><strong>PgBouncer (2 nodes)</strong>: Acts as a database connection pooler. It reduces the load on PostgreSQL by pooling numerous short-lived connections from applications. When used with HAProxy, it provides both accurate routing and connection pooling. For high availability, PgBouncer runs on both nodes and is accessed via HAProxy.</li><li><strong>Keepalived (2 nodes, alongside HAProxy)</strong>: Provides a single access point by assigning a virtual IP (VIP) between two HAProxy+PgBouncer servers. The active HAProxy server holds the VIP; if it fails, the VIP automatically migrates to the other HAProxy server. This ensures clients always connect through the same IP, offering uninterrupted service.</li><li><strong>pgAdmin (1 node, installed on one of the HAProxy/PgBouncer nodes)</strong>: A web-based interface for managing and monitoring the PostgreSQL cluster. While not critical, pgAdmin is a convenience tool for DBAs/DevOps. In this setup, pgAdmin is installed on one of the HAProxy nodes (preferably the one that usually holds the VIP).</li></ul><p><strong>Note</strong>: In this architecture, the terms <em>Primary</em> and <em>Replica</em> will be used. The Primary node is the only writable database node; Replicas are read-only nodes kept up to date via streaming replication from the Primary. Patroni automatically selects and manages the Primary node. Client connections are always routed to the Primary; if the Primary changes, routing is dynamically updated thanks to the collaboration of Patroni, Consul, and HAProxy.</p><p><strong>Servers and Sample IP Allocation</strong></p><p>As an example, we will use the following IP address plan. You can adapt it according to your own network configuration:</p><ul><li><strong>DB1</strong> (PostgreSQL + Patroni + Consul) — Hostname: db1, IP: 10.0.0.11</li><li><strong>DB2</strong> (PostgreSQL + Patroni + Consul) — Hostname: db2, IP: 10.0.0.12</li><li><strong>DB3</strong> (PostgreSQL + Patroni + Consul) — Hostname: db3, IP: 10.0.0.13</li><li><strong>Proxy1</strong> (HAProxy + PgBouncer + Keepalived + pgAdmin) — Hostname: proxy1, IP: 10.0.0.21</li><li><strong>Proxy2</strong> (HAProxy + PgBouncer + Keepalived) — Hostname: proxy2, IP: 10.0.0.22</li><li><strong>VIP</strong> (Keepalived Virtual IP) — 10.0.0.30 (shared virtual address for HAProxy/PgBouncer)</li></ul><p>It is important that all servers have synchronized system time (via NTP) and correctly configured name resolution (via DNS or /etc/hosts entries).</p><p>For example, each server’s hosts file can include the others as follows:</p><pre>10.0.0.11   db1<br>10.0.0.12   db2<br>10.0.0.13   db3<br>10.0.0.21   proxy1<br>10.0.0.22   proxy2<br>10.0.0.30   pgcluster-vip</pre><p><strong>Consul Installation and Configuration (DB1, DB2, DB3)</strong></p><p>First, we will install the Consul distributed lock store that will be used by Patroni on all three PostgreSQL nodes. Since Consul is critical for consistent leader election among Patroni nodes, we will set up a Consul cluster running in server mode on each of the DB servers.</p><ol><li><strong>Download and Install Consul</strong>:<br>It is recommended to install the latest version of Consul using HashiCorp’s official repository.<br>On each DB1, DB2, and DB3 server, follow these steps:</li></ol><pre># HashiCorp GPG key<br>sudo apt update<br>sudo apt install -y curl gnupg lsb-release<br>curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg<br><br># Add the HashiCorp repository to the APT sources list<br>echo &quot;deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] \<br>    https://apt.releases.hashicorp.com $(lsb_release -cs) main&quot; | \<br>    sudo tee /etc/apt/sources.list.d/hashicorp.list<br><br># Consul installation<br>sudo apt update<br>sudo apt install -y consul</pre><p>The steps above will install the consul package on the system. After the installation, a user will be created for Consul, and the default configuration directory will be located at /etc/consul.d.</p><p><strong>2. Configure Consul</strong>:<br>On each DB server, we will create a configuration file for Consul to run in server mode. Create the file /etc/consul.d/consul.hcl and configure it as shown below, using values specific to each server.</p><p>Example for DB1 /etc/consul.d/consul.hcl:</p><pre>datacenter  = &quot;dc1&quot;<br>node_name   = &quot;db1&quot;<br>bind_addr   = &quot;10.0.0.11&quot;         # The IP address of this server<br>client_addr = &quot;0.0.0.0&quot;           # Allow Consul UI and agent to listen on all interfaces<br>data_dir    = &quot;/var/lib/consul&quot;   # Data directory<br>log_level   = &quot;INFO&quot;<br><br>server          = true<br>bootstrap_expect= 3               # Total number of Consul servers in the cluster<br>retry_join      = [&quot;10.0.0.11&quot;, &quot;10.0.0.12&quot;, &quot;10.0.0.13&quot;]  # IP addresses of the other Consul servers<br><br># (Optional) Enable the web UI<br>ui_config {<br>  enabled = true<br>}</pre><p>Create the same file on <strong>DB2</strong> and <strong>DB3</strong> using their respective node_name and bind_addr values (db2/10.0.0.12, db3/10.0.0.13).<br> Make sure that all <strong>three IP addresses</strong> are included in the retry_join list (the list can be the same on all nodes). With these settings:</p><ul><li>Consul will run in <strong>server mode</strong> on each node (with leader election).</li><li>bootstrap_expect = 3 indicates that the cluster will form quorum with 3 servers.</li><li>retry_join ensures that each node knows how to reach the others and form the cluster.</li><li>client_addr = 0.0.0.0 allows the Consul agent API to be accessible from all interfaces (for better security, you can restrict it to your internal network).</li></ul><p><strong>3. Start the Consul service and check the status</strong>:<br>Once the configuration is complete, enable and start the Consul service on each DB server:</p><pre>sudo systemctl enable consul<br>sudo systemctl start consul</pre><p>During the initial startup, you can view the <strong>cluster membership messages</strong> in the Consul service logs (or via journalctl -u consul). After all three servers discover each other, one will be elected as the <strong>leader</strong>. You can check the cluster status from any node using the following command:</p><pre>consul members</pre><p>This command will list all Consul nodes, show whether they are <strong>alive</strong>, and indicate their <strong>roles</strong> (leader/follower).<br>If all members appear as <em>alive</em>, the Consul cluster has been successfully established.</p><p>Consul now provides a <strong>central service registry and distributed locking mechanism</strong> for Patroni.</p><p><strong>Note</strong>:<br>If you enabled the Consul web UI, you can also view the cluster status by navigating to http://&lt;dbX-ip&gt;:8500 in a browser.<br>(Access should ideally be done via a management network or a secure tunnel.)</p><p><strong>PostgreSQL and Patroni Installation (DB1, DB2, DB3)</strong></p><p>Now we will install <strong>PostgreSQL and Patroni</strong> on all three database servers.<br>Patroni is a Python application that controls PostgreSQL processes and manages the cluster by communicating with other Patroni instances via Consul. Patroni takes over PostgreSQL’s configuration, startup, and shutdown processes. Therefore, <strong>we will run Patroni as the main service</strong>, not the PostgreSQL service itself.</p><p><strong>Installing PostgreSQL (Ubuntu 22.04)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L_Sp4Evjhe7yzrmikf7UNg.png" /><figcaption>Posgresql</figcaption></figure><p>Install the PostgreSQL server package on each of the DB1, DB2, and DB3 servers:</p><pre>sudo apt install -y postgresql postgresql-contrib</pre><p>After the installation, Ubuntu will automatically start a default PostgreSQL cluster (14/main). Since we will manage our own cluster using Patroni, we need to stop and disable the default PostgreSQL service:</p><pre>sudo systemctl stop postgresql<br>sudo systemctl disable postgresql</pre><p>If necessary, you can also delete the default PostgreSQL data directory<br>(usually located at /var/lib/postgresql/14/main).When Patroni starts for the first time, it will initialize its own data directory.</p><p><strong>Note</strong>:<br>Patroni must be able to find PostgreSQL binaries (like initdb, pg_ctl, etc.) in the system PATH. On Ubuntu, these are located under /usr/lib/postgresql/14/bin/. One way to make them accessible is to symlink them into /usr/bin:</p><pre>sudo ln -s /usr/lib/postgresql/14/bin/* /usr/bin/</pre><p>This allows Patroni to locate the required executables at runtime. Alternatively, you can set the PATH explicitly in the Patroni <strong>systemd unit file</strong> (as will be shown below).</p><p><strong>Installing Patroni</strong></p><p>Patroni is a Python-based tool and can be installed using pip. We will install Patroni with <strong>Consul support</strong>. On each DB1, DB2, and DB3 node, follow the steps below:</p><pre>sudo apt install -y python3-pip python3-dev libpq-dev python3-psycopg2</pre><pre>sudo pip3 install patroni[consul]</pre><p>This command will install Patroni and the required Python module (python-consul) to enable integration with Consul.</p><p>If this command fails with a psycopg2 error, it might mean the APT-installed psycopg2 didn’t take effect. In that case, you can also run:</p><pre>sudo pip3 install psycopg2-binary</pre><p>Verify that the installation was successful:</p><pre>patroni -h</pre><p>If this command returns the help output, Patroni has been installed successfully.</p><p>Each node running Patroni must have a YAML configuration file (e.g., /etc/patroni.yml).<br> This file will be mostly identical across all nodes, but values such as <strong>node name</strong>, <strong>IP addresses</strong>, and a few others will be adjusted per node.</p><p><strong>Creating Patroni Configuration File</strong></p><p>Below is an example <strong>Patroni configuration file</strong>:</p><pre>scope: pgcluster            # Cluster name (must be the same on all nodes)<br>namespace: /service/        # Prefix to be used in Consul KV (default is &quot;/service/&quot;)<br>name: db1                   # This node&#39;s name (must be different on each server)<br><br>restapi:<br>  listen: 0.0.0.0:8008<br>  connect_address: 10.0.0.11:8008   # API address for this node, accessible from other nodes<br><br>consul:<br>  host: 127.0.0.1<br>  port: 8500<br><br>bootstrap:<br>  # Settings used during initial cluster bootstrap<br>  dcs:<br>    ttl: 30<br>    loop_wait: 10<br>    retry_timeout: 10<br>    maximum_lag_on_failover: 1048576   # (1 MB) Max lag allowed for failover<br>    postgresql:<br>      use_pg_rewind: true             # Use pg_rewind if data is behind<br>      use_slots: true                 # Use replication slots to prevent data loss<br>      parameters:<br>        max_connections: 100<br>        shared_buffers: 256MB<br>        max_wal_senders: 10<br>        wal_level: replica<br>        hot_standby: &quot;on&quot;<br>        hot_standby_feedback: &quot;on&quot;<br>        # Additional PostgreSQL parameters can be defined here (applies to all nodes)<br><br>  initdb:<br>    - encoding: UTF8<br>    - locale: tr_TR.UTF-8   # Sample locale, change as needed<br>    - data-checksums<br><br>  # Set initial passwords when the cluster is created:<br>  users:<br>    postgres:<br>      password: &quot;YourPostgresSuperUserPassword&quot;<br>      options:<br>        - createrole<br>        - createdb<br>    repl:<br>      password: &quot;YourReplicationPassword&quot;<br>      options:<br>        - replication<br><br>postgresql:<br>  listen: 0.0.0.0:5432<br>  connect_address: 10.0.0.11:5432     # This node&#39;s PostgreSQL address, accessible from other nodes<br>  data_dir: /var/lib/postgresql/14/main<br>  config_dir: /etc/postgresql/14/main  # Optional: PG config path, managed by Patroni<br>  bin_dir: /usr/lib/postgresql/14/bin  # PostgreSQL binary path (needed if not in PATH)<br>  authentication:<br>    replication:<br>      username: repl<br>      password: &quot;YourReplicationPassword&quot;<br>    superuser:<br>      username: postgres<br>      password: &quot;YourPostgresSuperUserPassword&quot;</pre><p>Let’s briefly explain the above file:</p><ul><li><strong>scope</strong>: A unique name for the cluster. It must be the same on all nodes. This name will also be used in keys within Consul (combined with the namespace).</li><li><strong>namespace</strong>: Determines under which path Patroni will store records in Consul. The default /service/ is sufficient. (Consul KV entries will be stored under /service/pgcluster/.)</li><li><strong>name</strong>: The name of this Patroni node. It must be unique per server (e.g., db1, db2, db3).</li><li><strong>restapi</strong>: Patroni provides a REST API on each node to report its status.</li><li>listen: The IP:port the API will listen on.</li><li>connect_address: The address other cluster members will use to reach this node.<br> Use 0.0.0.0:8008 for listen on all servers, but connect_address must be each node’s own IP.</li><li><strong>consul</strong>: Specifies the DCS (Distributed Configuration Store) connection Patroni will use.<br> Since the Consul agent runs locally on port 8500 on each server, we specify host: 127.0.0.1 and port: 8500.<br> (If you use Consul ACLs, you can add a token and other parameters here, but we keep it simple.)</li><li><strong>bootstrap</strong>:<br> Contains settings applied only when the cluster is first created.<br> After bootstrap, Patroni saves some of these settings in Consul and reuses them later.</li><li><strong>dcs</strong>: Settings stored in Consul for the entire cluster.</li><li>ttl: Time-to-live for the leadership lock (e.g., 30s).</li><li>loop_wait: Time between checks (e.g., 10s).</li><li>retry_timeout: Timeout when retrying failed operations.</li><li>maximum_lag_on_failover: Max WAL lag allowed before failover.</li><li><strong>use_pg_rewind: true</strong>: Enables automatic sync via pg_rewind if a node is behind (e.g., if the old primary rejoins).</li><li><strong>use_slots: true</strong>: Uses replication slots to prevent WAL loss if replicas fall behind.</li><li><strong>parameters</strong>:<br> These are PostgreSQL’s postgresql.conf parameters and will be applied on all nodes.<br> Some important ones shown:</li><li>wal_level: replica, max_wal_senders, and hot_standby are essential for replication.</li><li><strong>initdb</strong>:<br> If the data_dir is empty, Patroni will run initdb using these flags.<br> You can define encoding, locale, and enable data-checksums (useful for detecting corruption, optional).</li><li><strong>users</strong>:<br> Defines the PostgreSQL users and their passwords during the initial cluster bootstrap.<br> We define a superuser (postgres) and a replication user (repl) with appropriate roles.<br> These users are created automatically when initdb is run.</li><li><strong>postgresql</strong>:<br> Defines how Patroni will manage the PostgreSQL instance.</li><li><strong>listen</strong>: The IP:port PostgreSQL will listen on.<br> Usually 0.0.0.0:5432 to allow both local and network access.</li><li><strong>connect_address</strong>:<br> The address and port other nodes (and clients like HAProxy) will use to connect to this PostgreSQL instance.<br> This should be the node&#39;s actual IP (e.g., 10.0.0.11:5432 for db1).</li><li><strong>data_dir</strong>:<br> The data directory for PostgreSQL. For APT installations, this is typically /var/lib/postgresql/14/main.<br> This directory might already be initialized by APT; you may clear it manually.<br> Patroni will run initdb on the first node and use pg_basebackup on the others.</li><li><strong>config_dir</strong>:<br> Directory for PostgreSQL config files, usually /etc/postgresql/14/main.<br> Patroni will manage postgresql.conf, pg_hba.conf, etc. in this location.<br> (Optional but recommended for clarity.)</li><li><strong>bin_dir</strong>:<br> Path to PostgreSQL binaries.<br> If you haven&#39;t added them to PATH, this ensures Patroni can find pg_ctl, initdb, etc.<br> For PostgreSQL 14 on Ubuntu 22.04, this is /usr/lib/postgresql/14/bin.</li><li><strong>authentication</strong>:<br> Credentials for Patroni to connect to PostgreSQL as the superuser and replication user.<br> These should match the credentials defined in bootstrap.users.<br> Patroni uses these to set up replication and perform superuser operations.</li></ul><h3>Important Notes:</h3><ul><li>Create this configuration file on all three servers.</li><li>Adjust the following per server:</li><li>name</li><li>restapi.connect_address</li><li>postgresql.connect_address</li><li>All other parts (scope, namespace, etc.) should remain identical across nodes.</li></ul><p><strong>Security Note</strong>:<br> In this example, passwords are stored in plain text in the config file.<br> In production, you can use environment variables or .pgpass for storing credentials securely.<br> Make sure to restrict file permissions with:</p><pre>chmod 600 /etc/patroni.yml</pre><p>Let me know if you’d like the adapted config for db2 and db3, or if we should move on to setting up the systemd unit file for Patroni.</p><p><strong>Systemd service description for Patroni</strong></p><pre>ini<br>CopyEdit<br>[Unit]<br>Description=PostgreSQL HA cluster node Patroni<br>After=network.target<br>Wants=network-online.target<br><br>[Service]<br>Type=simple<br>User=postgres<br>Group=postgres<br>Environment=&quot;PATH=/usr/lib/postgresql/14/bin:/usr/bin:/bin&quot;<br>ExecStart=/usr/local/bin/patroni /etc/patroni.RML<br>ExecReload=/bin/Kill -s HUP $MAINPID<br>KillMode=process<br>Restart=on-failure<br>LimitNOFILE=1048576<br><br>[Install]<br>WantedBy=multi-user.target</pre><pre>--<br>sudo systemctl enable patroni<br>sudo systemctl start patroni</pre><p><strong>HAProxy setup and configuration (proxy1, proxy2)</strong></p><p>HAProxy will route to the correct node in a cluster, client connections that now we will build layer. We have two HAProxy instance (proxy1 and proxy2 on servers), these will share with keepalived virtual IP (see below). HAProxy, which is the primary node that provides a REST API using Node Patroni each of which is a replica of the links that will understand and will route accordingly.</p><ol><li><strong>Installation</strong></li></ol><pre><br>sudo apt install-y haproxy</pre><p><strong>2. Create the HAProxy configuration:</strong> default config file /etc/haproxy/haproxy.cfg is on the way. This is the end of file (or an appropriate substitute) we will add the required definitions for PostgreSQL. Existing settings to the defaults, dropping just at the bottom we show the parts we need:</p><p><strong>/etc/haproxy/haproxy.cfg</strong></p><pre># Optional: HAProxy global/defaults or stats interface config could go above<br><br># Frontend for PostgreSQL Write Traffic (Primary Node)<br>frontend pg_rw<br>    bind *:5000<br>    mode tcp<br>    default_backend pg_primary<br><br># Backend for PostgreSQL Primary Node<br>backend pg_primary<br>    mode tcp<br>    option httpchk OPTIONS /master<br>    http-check expect status 200<br>    server db1 10.0.0.11:5432 check port 8008 inter 3s fall 3 rise 2 on-marked-down shutdown-sessions<br>    server db2 10.0.0.12:5432 check port 8008 inter 3s fall 3 rise 2 on-marked-down shutdown-sessions<br>    server db3 10.0.0.13:5432 check port 8008 inter 3s fall 3 rise 2 on-marked-down shutdown-sessions<br><br># Frontend for PostgreSQL Read-Only Traffic (Replicas)<br>frontend pg_ro<br>    bind *:5001<br>    mode tcp<br>    default_backend pg_replicas<br><br># Backend for PostgreSQL Replica Nodes<br>backend pg_replicas<br>    mode tcp<br>    option httpchk OPTIONS /replica<br>    http-check expect status 200<br>    server db1 10.0.0.11:5432 check port 8008 inter 3s fall 3 rise 2 on-marked-down shutdown-sessions<br>    server db2 10.0.0.12:5432 check port 8008 inter 3s fall 3 rise 2 on-marked-down shutdown-sessions<br>    server db3 10.0.0.13:5432 check port 8008 inter 3s fall 3 rise 2 on-marked-down shutdown-sessions<br><br># Optional: HAProxy Statistics Interface (for monitoring)<br># listen stats<br>#     bind *:7000<br>#     stats enable<br>#     stats uri /stats<br>#     stats auth admin:yourpassword</pre><ul><li><strong>pg_rw frontend (port 5000): </strong>This frontend listens on port 5000 for incoming database traffic. All PostgreSQL clients should connect to this port for <strong>read-write</strong> operations. We use mode tcp because HAProxy doesn’t inspect or interpret the PostgreSQL protocol—it simply forwards raw TCP traffic. This frontend is connected to the pg_primary backend, which ensures clients always reach the current <strong>primary</strong> node.</li><li><strong>pg_primary backend:</strong> This backend contains all three PostgreSQL nodes (db1, db2, db3), but only the current leader (primary) is considered healthy. HAProxy will direct traffic only to the node that passes the health check. option httpchk OPTIONS /master and http-check expect status 200:These lines configure HAProxy to send an HTTP OPTIONS /master request to the REST API of each node (port 8008). Patroni responds with: 200 OK <strong>only</strong> on the current <strong>leader</strong>, 503 Service Unavailable on all replicas. This ensures HAProxy routes write traffic only to the leader. inter 3s fall 3 rise 2: Health checks are made every <strong>3 seconds</strong>. If a node <strong>fails 3 times in a row</strong>, it is marked as <strong>down</strong>. If it then passes <strong>2 times consecutively</strong>, it is marked as <strong>up</strong> again. This results in about <strong>9 seconds</strong> of detection time before failover takes place. on-marked-down shutdown-sessions:When a node is marked down (e.g., an old primary is no longer available), this option ensures <strong>active client sessions are forcibly closed</strong>. Clients will then reconnect—automatically establishing new sessions with the new leader.</li><li><strong>In summary</strong>: Only one node (the current leader) will be active in this backend at any given time. This guarantees that <strong>write requests never reach replicas by mistake</strong>.</li><li><strong>pg_ro frontend (port 5001): </strong>This optional frontend listens on port 5001 for <strong>read-only</strong> traffic. It connects to the pg_replicas backend, which is intended to <strong>distribute read load across replicas</strong>. If your application cannot separate read and write queries, this port may go unused.</li><li><strong>pg_replicas backend: </strong>Similar to the primary backend but in reverse logic: Uses health checks to /replica on port 8008. Only <strong>replica (standby)</strong> nodes will return 200 OK. The current <strong>primary node</strong> responds with 503, and is marked unhealthy in this context. If you have multiple replicas, all will be active and HAProxy will use <strong>round-robin</strong> balancing. Optional: You can define separate ports for <strong>sync</strong> and <strong>async</strong> replicas if fine-grained traffic control is needed.</li><li><strong>Unified vs. Split Access Ports:</strong><br>You can either: Connect all traffic (read and write) to <strong>port 5000</strong> — HAProxy will send everything to the primary. Or, split access — write operations to <strong>port 5000</strong>, read operations to <strong>port 5001</strong> — if your application supports this.</li><li><strong>HAProxy Statistics Interface (optional): </strong>You can enable a web-based stats dashboard by adding: This provides a live view of backend status, connection counts, and failover behavior. <strong>Security tip</strong>: Always protect this page with a strong password.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W29InQZVn9UjKMb2zTzq-A.png" /></figure><p><strong>Installation and configuration Pooler (proxy1, proxy2)</strong></p><p>Pooler, lightweight connection pool for PostgreSQL server. The minimum number of client connections to the database connection by turning many of the efficiency of the system improves. Our clients work directly on the proxy server and one shall be pgbouncer senaryomiz both in Pooler. Pooler via HAProxy will reach the correct database in the background. In this way:</p><ul><li>Client-side <strong>connection pooling</strong> will be provided (pooler for PostgreSQL as opening and closing the application havuzladig he wouldn’t have to fork a new process each time).</li><li>With HAProxy be integrated to Pooler will always connect to the correct node; failover in the event of re-establishing open connections will continue.</li></ul><ol><li><strong>Pgbouncer set up:</strong> both on the proxy server:</li></ol><pre>sudo apt install-y Pooler</pre><p>This process Pooler creates the user and the service, but Debian/Ubuntu on <strong>default Pooler does not start automatically</strong>.</p><p><strong>2. The default settings enable:</strong> /etc/default/Pooler file editor open and start=0 with the line Start=1 do. We are thus enabled to begin service systemd. Also PGBOUNCER_USER=Pooler and PGBOUNCER_GROUP=Pooler as you can leave the (default).</p><p><strong>3. Pooler edit the configuration:</strong> main configuration file is /etc/Pooler/Pooler.the download is on it’s way. Examples include the default file.</p><p><strong>etc/Pooler/Pooler.ini</strong></p><pre>[databases]<br># Clients that will access the database definitions.<br># Format: logical_db_name = host:port dbname=actual_db_name pool_size=...<br># Suppose we have a database named &quot;myapp&quot; that the application will use.<br>myapp = host=127.0.0.1 port=5000 dbname=myapp pool_size=20<br><br># Optional: read-only traffic routed to replicas<br>myapp_ro = host=127.0.0.1 port=5001 dbname=myapp pool_size=20<br><br>[pgbouncer]<br>listen_addr = 0.0.0.0<br>listen_port = 5432<br><br># Authentication and pool settings<br>auth_type = md5<br>auth_file = /etc/pgbouncer/userlist.txt<br>pool_mode = session<br>max_client_conn = 500Explanation:</pre><ul><li>PgBouncer listens on port 5432 for incoming client connections.</li><li>It forwards requests to local HAProxy frontends (5000 for primary, 5001 for replicas).</li><li>myapp_ro is optional and used if the application separates read vs write connections.</li><li>pool_mode = session allows each client a dedicated connection for the life of their session.</li></ul><p>Create /etc/pgbouncer/userlist.txt with database credentials:</p><pre>&quot;myappuser&quot; &quot;md5c2fcec2d60fd538e6294a7312d6237b7&quot;</pre><pre>echo -n &#39;Sifre123myappuser&#39; | md5sum</pre><pre>sudo systemctl start pgbouncer<br>sudo systemctl enable pgbouncer</pre><p>For Testing</p><pre>psql -h 127.0.0.1 -p 5432 -U myappuser myapp</pre><p><strong>Keepalived VIP Configuration (proxy1, proxy2)</strong></p><p>Because it is our proxy server with two clients connected to them we need to figure out how. <strong>Keepalived</strong> virtual IP address between two servers by using we’ll hand over. Thus, the applications will be bound to the same IP; the IP will belong to this active proxy in the background. Proxy IP address automatically active when the other falls, and the clients will pass to seamless communication will continue via the same IP.</p><p><strong>Installing Keepalived</strong></p><pre>sudo apt install -y keepalived</pre><p><strong>Configuration File (Master)</strong></p><pre>vrrp_script chk_services {<br>    script &quot;/bin/pidof haproxy &amp;&amp; /bin/pidof pgbouncer&quot;<br>    interval 2<br>    weight 2<br>}<br><br>vrrp_instance VI_PG {<br>    state MASTER<br>    interface ens160  # replace with your network interface name<br>    virtual_router_id 51<br>    priority 101<br>    advert_int 1<br><br>    authentication {<br>        auth_type PASS<br>        auth_pass 12345<br>    }<br><br>    virtual_ipaddress {<br>        10.0.0.30/24 dev ens160<br>    }<br><br>    track_script {<br>        chk_services<br>    }<br>}</pre><p><strong>Configuration File (Backup)</strong></p><pre>vrrp_script chk_services {<br>    script &quot;/bin/pidof haproxy &amp;&amp; /bin/pidof pgbouncer&quot;<br>    interval 2<br>    weight 2<br>}<br><br>vrrp_instance VI_PG {<br>    state BACKUP<br>    interface ens160<br>    virtual_router_id 51<br>    priority 100<br>    advert_int 1<br><br>    authentication {<br>        auth_type PASS<br>        auth_pass 12345<br>    }<br><br>    virtual_ipaddress {<br>        10.0.0.30/24 dev ens160<br>    }<br><br>    track_script {<br>        chk_services<br>    }<br>}</pre><ul><li>vrrp_instance defines the shared VIP configuration.</li><li>priority determines which node is active (higher wins).</li><li>chk_services ensures HAProxy and PgBouncer are healthy before assigning VIP.</li><li>auth_pass and virtual_router_id must match on both nodes.</li></ul><pre>sudo systemctl enable keepalived<br>sudo systemctl start keepalived</pre><p>Check the VIP on the MASTER:</p><pre>ip addr show ens160</pre><p>The VIP 10.0.0.30 should be bound on the active node only. .Test failover by stopping HAProxy or PgBouncer on the MASTER node and watching the VIP move to BACKUP.</p><p><strong>Pgadmin Installation</strong></p><pre>sudo apt install -y curl ca-certificates gnupg<br>curl https://www.pgadmin.org/static/packages_pgadmin_org.pub | sudo apt-key add -<br>sudo sh -c &#39;echo &quot;deb https://ftp.postgresql.org/pub/pgadmin/pgadmin4/apt/ubuntu jammy pgadmin4 main&quot; &gt; /etc/apt/sources.list.d/pgadmin4.list&#39;<br>sudo apt update<br>sudo apt install -y pgadmin4-web</pre><p>Configure: sudo /usr/pgadmin4/bin/setup-web.sh</p><p>Access: <a href="http://10.0.0.21/pgadmin4">http://10.0.0.21/pgadmin4</a></p><p>Add a new server using:</p><ul><li>Host: 10.0.0.30</li><li>Port: 5432</li><li>Username: postgres</li><li>Password: Your set password</li></ul><p>You’ll be able to see and manage the PostgreSQL cluster from the UI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mt3ZR3tp1UbV5miCTAFLUw.png" /></figure><p><strong>Conclusion and final checks</strong></p><p>With these steps, 3-node (1 Primary + 2 Replica) PostgreSQL cluster to be managed by Patroni we was established. Two HAProxy+Pooler with a single virtual server can be connected via IP to the cluster has become clients. In this architecture <strong>, each component of backup available</strong> , because even though a failure at any point, the system continues to stay on his feet in a completely automatic way<a href="https://medium.com/@mert_yavasca/keep-your-postgres-stable-ha-architecture-afa3b68c1c02#:~:text=you%20can%20leverage%20the%20latest,failover%20between%20HA%20Proxy%20servers">medium.com</a>. To summarize:</p><ul><li><strong>Automatic failover:</strong> if the primary database node is interrupted, Patroni and HAProxy will detect this and automatically within a few seconds directs all traffic to the primary replica makes new bind<a href="https://medium.com/@mert_yavasca/keep-your-postgres-stable-ha-architecture-afa3b68c1c02#:~:text=1,change%20parameters%20for%20all%20nodes">medium.com</a>​<a href="https://www.percona.com/blog/haproxy-patroni-setup-using-health-check-endpoints-and-debugging/#:~:text=bind%20,maxconn%20100%20check%20port%208008">percona.com</a>. The application side, only a short interruption (disconnect and reconnect) is.</li><li><strong>Load balancing (read Scaling):</strong> optionally, the HAProxy configuration read requests (port 5001) we can distribute to replica. Thus, reporting, analytics, such as read-intensive operations can be done without fraying bind. Just goes to bind processes that require writing.</li><li><strong>Connection pool:</strong> connection Pooler thanks to hundreds of client controlled even with as many links as this is reflected in PostgreSQL. Mikroservis especially in architecture, each service has its own database instead of managing a pool of connections makes this center Pooler. Discover new connections and Failover in case of breaks in the background Pooler directs you to the right place (if necessary, open the dial-up connection).</li><li><strong>Single access point:</strong> we provide with VIP Keepalived our system completely from the applications that use the internal details of the cluster that they’re abstract. Applications should always have the same IP:port which connects to the passive server in the back they don’t have to know which ones are active.</li><li><strong>Management and monitoring:</strong> Patroni REST API presents the status of each node. If you want in on this patronictl Grafa to or integrations you can build. you can easily manage the database with tools such as pgAdmin. Commands interface or the cluster with Consul records (Kim, the replica list, etc.) you can see. HAProxy stats interface (if enabled) keeps track of the distribution of instant connectivity.</li></ul><p><strong>Finally</strong>, the configuration files of each component to ensure that the details are correct and don’t forget to follow. Especially the first setup:</p><ul><li>Patroni logs (journalctl -u patroni) — Consul connection problems, replication errors, etc., can be seen from here.</li><li>PostgreSQL logs (/var/log/postgresql/postgresql-14-main.log or journalctl) — if you have errors when starting with pg_ctl Patroni will appear here.</li><li>HAProxy logs (/var/log/haproxy.log in or rsyslog usually via) — Health, check the status of the backend up/down messages can be seen.</li><li>Pooler log (/var/log/Pooler/Pooler.log) — successful connections, faulty initiatives, etc.</li><li>Keepalived log (journalctl -u keepalived) — take VIP changes here.</li></ul><p>The structure of this high-availability architecture is proven in real-life environments. Patroni+Consul leadership, management, and <strong>automatic failover</strong> is provided in a fast and reliable manner. HAProxy is conducted transparently with client redirection. Keepalived with proxy layer redundancy is completed. Ultimately, the <strong>single-point-failure (SPOF) without</strong> infrastructure, you will have a PostgreSQL<a href="https://medium.com/@mert_yavasca/keep-your-postgres-stable-ha-architecture-afa3b68c1c02#:~:text=you%20can%20leverage%20the%20latest,failover%20between%20HA%20Proxy%20servers">medium.com</a>. Now both secured and your database performance, both treatments can work in a way that won’t make you feel the disruptions to the outside world that is.</p><p><strong>Sources:</strong><br><a href="https://medium.com/@mertyavasca/keep-your-postgres-stable-ha-architecture-4fbb22426fdb">https://medium.com/@mertyavasca/keep-your-postgres-stable-ha-architecture-4fbb22426fdb</a></p><p><a href="https://www.percona.com/blog/haproxy-patroni-setup-using-health-check-endpoints-and-debugging">https://www.percona.com/blog/haproxy-patroni-setup-using-health-check-endpoints-and-debugging</a></p><p><a href="https://patroni.readthedocs.io/en/latest/installation.html">https://patroni.readthedocs.io/en/latest/installation.html</a><strong> </strong><a href="https://github.com/jayakrishnanl/PostgresPatroniCluster">https://github.com/jayakrishnanl/PostgresPatroniCluster</a></p><p><a href="https://snapshooter.com/learn/how-to-setup-postgresql-cluster-using-patroni-ubuntu-20-04">https://snapshooter.com/learn/how-to-setup-postgresql-cluster-using-patroni-ubuntu-20-04</a></p><p><a href="https://www.experts-exchange.com/questions/29165468/Patroni-Haproxy-and-pg-bouncer-architecture.html">https://www.experts-exchange.com/questions/29165468/Patroni-Haproxy-and-pg-bouncer-architecture.html</a></p><p><a href="https://stackoverflow.com/questions/57485747/how-to-get-pgbouncer-auto-start-on-reboot-on-linux">Scroll until next child visible, Javascript and JQuery</a></p><h3>Thank you for being a part of the community</h3><p><em>Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://x.com/inPlainEngHQ"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/@InPlainEnglish"><strong>YouTube</strong></a> | <a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a> | <a href="https://open.spotify.com/show/7qxylRWKhvZwMz2WuEoua0"><strong>Podcast</strong></a> | <a href="https://differ.blog/inplainenglish"><strong>Differ</strong></a> | <a href="https://twitch.tv/inplainenglish"><strong>Twitch</strong></a></li><li><a href="https://cofeed.app/"><strong>Check out CoFeed, the smart way to stay up-to-date with the latest in tech</strong></a> <strong>🧪</strong></li><li><a href="https://differ.blog/"><strong>Start your own free AI-powered blog on Differ</strong></a> 🚀</li><li><a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Join our content creators community on Discord</strong></a> 🧑🏻‍💻</li><li>For more content, visit <a href="https://plainenglish.io/"><strong>plainenglish.io</strong></a> + <a href="https://stackademic.com/"><strong>stackademic.com</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42e37f27e383" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/high-avalaible-monolith-posgresql-cluster-with-patroni-consul-haproxy-42e37f27e383">High Avalaible Monolith Posgresql Cluster with Patroni&amp;Consul&amp;Haproxy</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>