<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jonathan Duran on Medium]]></title>
        <description><![CDATA[Stories by Jonathan Duran on Medium]]></description>
        <link>https://medium.com/@jon-duran?source=rss-89051b7c318f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*x15UzvjV6RFJL8l051dK4w.png</url>
            <title>Stories by Jonathan Duran on Medium</title>
            <link>https://medium.com/@jon-duran?source=rss-89051b7c318f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 07:36:05 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jon-duran/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Validate Smarter at the Row Level: A Four-Layer Approach]]></title>
            <link>https://blog.dataengineerthings.org/validate-smarter-at-the-row-level-a-four-layer-approach-c6aac0f752dd?source=rss-89051b7c318f------2</link>
            <guid isPermaLink="false">https://medium.com/p/c6aac0f752dd</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-architecture]]></category>
            <dc:creator><![CDATA[Jonathan Duran]]></dc:creator>
            <pubDate>Sun, 03 May 2026 04:08:46 GMT</pubDate>
            <atom:updated>2026-05-03T05:04:03.053Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>A typical data validation flow goes through four phases: ingesting raw data, validating at the row level, validating at the dataset level, and finally publishing “good” data while quarantining “bad” data. This article focuses on the second phase — row-level validation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*83IM1koxgr0Tb0AI9d1cjA.png" /></figure><p>This is the first point where I actually start validating data after it’s been ingested in its raw form from an upstream source. At this stage, I am looking at individual records and asking: <em>does this row make sense on its own?</em></p><p>I like to think about row-level validation in four layers:</p><ul><li><strong>Schema validation</strong> → does the data match the expected structure (types, required fields, nullability)?</li><li><strong>Value formats</strong> → are the values correctly structured (e.g., formats like state codes, IDs)?</li><li><strong>Business rules</strong> → do fields make sense together (e.g., timestamps, status vs. amounts)?</li><li><strong>Metrics &amp; entities</strong> → do the fields that power key metrics and definitions hold up?</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/446/1*N1WJ0JCUiHoR_o-QLbwTbA.png" /></figure><p>The challenge is deciding how strict to be. Too strict, and you risk breaking pipelines or creating stale data over minor issues. Too lax, and bad data slips through and causes problems downstream. The goal isn’t to validate everything — it’s to apply the right level of validation where it actually matters.</p><p>To make this concrete, I’ll use a simple data contract and implement these layers using Pydantic. It’s a great fit for this stage because it allows you to express validation rules directly alongside your schema, making it easy to reason about what’s being enforced and why.</p><p>Here is our sample data contract:</p><pre>          +------------------+------------+----------+----------+<br>          | Field Name       | Data Type  | Nullable | Required |<br>          +------------------+------------+----------+----------+<br>          | id               | string     | No       | Yes      |<br>          | name             | string     | Yes      | No       |<br>          | description      | string     | Yes      | No       |<br>          | state            | string     | No       | Yes      |<br>          | payment_amount   | float      | No       | Yes      |<br>          | refund_amount    | float      | No       | Yes      |<br>          | invoice_status   | string     | No       | Yes      |<br>          | created_at       | datetime   | No       | Yes      |<br>          | last_updated     | datetime   | No       | Yes      |<br>          +------------------+------------+----------+----------+</pre><h3><strong>Layer 1 — Schema Enforcement</strong></h3><p>The first layer of validation is schema enforcement. It’s usually the most straightforward to implement because — assuming the data producer has provided documentation or a contract — you have a clear definition of what’s being promised: what fields exist, which are required, their data types, and whether they can be null.</p><p>At this stage, I’m translating that contract directly into code.</p><p>Using our sample data contract, I can model it in Pydantic like this:</p><pre>from datetime import datetime<br><br>from pydantic import BaseModel<br><br><br>class ApiResponse(BaseModel):<br>    id: str<br>    name: str | None = None<br>    description: str | None = None<br>    state: str<br>    payment_amount: float<br>    refund_amount: float<br>    invoice_status: str<br>    created_at: datetime<br>    last_updated: datetime</pre><p>At this layer, the checks are almost entirely defined by the agreement between the data engineer and the data producer. Whether that agreement comes in the form of API specs, contracts, or public documentation, it dictates the field constraints — and ultimately the checks — I should apply.</p><p>There’s nothing fancy happening here. I’m working with a limited set of basic checks — required vs. optional fields, data types, and nullability — which makes it easy to reason about what actually needs to be enforced.</p><p>Because of that, this layer is also where I make conscious decisions about how strict to be. If a field is documented as an integer but occasionally shows up as a numeric string, I don’t necessarily want to fail the pipeline over that. In many cases, it’s better to accept the value and coerce it into the expected type rather than risk stale pipelines over a minor inconsistency.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DkB-bc5DFRs-a9JbHTkktA.png" /></figure><p>The goal here isn’t perfection — it’s establishing a reliable baseline. I want to make sure the data has the shape I expect, without introducing unnecessary fragility this early in the validation process.</p><h3>Layer 2 — Value Formats</h3><p>The next layer is value format validation. Instead of just asking <em>“is this a string?”</em>, I start asking <em>“is this the right kind of string?”</em></p><p>This layer is a step up in complexity, but not by much. I’m still mostly working off of the same contract, but now I may also need input from analysts or analytics engineers to understand which formats actually matter downstream.</p><p>For example, say the contract tells me a state field is provided as a postal abbreviation (NY, CA, TX).</p><p>I can enforce that expectation directly:</p><pre>from datetime import datetime<br>from typing import Annotated<br><br>from pydantic import BaseModel, Field<br><br># Custom field that enforces postal abbreviated formats<br>PostalCodeAbbreviatedState = Annotated[<br>    str,<br>    Field(pattern=&quot;^(AL|AK|AZ|...|WY)$&quot;)<br>]<br><br><br>class ApiResponse(BaseModel):<br>    id: str<br>    name: str | None = None<br>    description: str | None = None<br>    state: PostalCodeAbbreviatedState<br>    payment_amount: float<br>    refund_amount: float<br>    invoice_status: str<br>    created_at: datetime<br>    last_updated: datetime</pre><p>This is still grounded in the same idea as schema validation — we’re enforcing what the producer told us — but we’re now validating <em>formats</em> within those types.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JMjRRmPK-GeoN1RCplKFbQ.png" /></figure><p>That said, this is also where the risk of unnecessary checks starts to creep in. Just because I <em>can</em> validate that a state is in abbreviated form doesn’t mean I <em>should</em>. If there isn’t a single transformation, dimension, metric, or entity definition that depends on that format, I’m adding fragility without real value. And when that check inevitably fails, I’m left explaining why a table is stale over something that didn’t actually matter.</p><p>So while this layer is useful, it’s also where I start being more selective. Not every possible check is worth implementing.</p><h3>Layer 3 — Business Rules</h3><p>The business rules layer is where validation starts to move beyond the producer’s schema. A business rule is a check that validates whether values make sense in the context of how the business actually operates.</p><p>At this stage, I’m no longer relying only on the data producer. This is where I need more input from analysts, analytics engineers, or other people close to the business to understand which checks are actually valuable.</p><p>A lot of these checks involve multiple fields that need to make sense together. For example, if an invoice has a status of DRAFT, I would expect a field like paid_amount to be either null or 0.</p><p>In this example, I can define an enum for invoice_status to restrict it to valid values, and then enforce that those values align logically with the paid_amount for each invoice.</p><pre>from datetime import datetime<br>from enum import StrEnum, auto<br>from typing import Annotated, Self<br><br>from pydantic import BaseModel, Field<br><br><br># Define all possible values for invoice_status<br>class InvoiceStatus(StrEnum):<br>  DRAFT = auto()<br>  PENDING = auto()<br>  PAID = auto()<br>  CANCELLED = auto()<br><br><br>PostalCodeAbbreviatedState = Annotated[<br>    str,<br>    Field(pattern=&quot;^(AL|AK|AZ|...|WY)$&quot;)<br>]<br><br>class ApiResponse(BaseModel):<br>    id: str<br>    name: str | None = None<br>    description: str | None = None<br>    state: PostalCodeAbbreviatedState<br>    payment_amount: float<br>    refund_amount: float<br>    invoice_status: InvoiceStatus<br>    created_at: datetime<br>    last_updated: datetime<br>    <br>    # Validate that invoice_status and paid_amount values are aligned.<br>    @model_validator(mode=&quot;after&quot;)<br>    def validate_paid_amount_for_status(self) -&gt; Self:<br>        if self.invoice_status in {&quot;draft&quot;, &quot;pending&quot;}:<br>            if self.paid_amount not in {None, Decimal(&quot;0.00&quot;)}:<br>                raise ValueError(<br>                    &quot;paid_amount must be 0.00 or null when invoice_status is DRAFT or PENDING&quot;<br>                )<br>        return self</pre><p>Keep in mind, using an enum to validate a field is very similar to defining an annotated type with additional constraints.</p><p>The point of each layer isn’t the specific implementation — it’s what’s driving the need for the check. The layers of validation are defined more by the context, the intent, and the people involved than by the exact logic itself.</p><p>In the next section, you’ll see similar checks applied again, but for entirely different reasons.</p><h3>Layer 4 — Metrics And Entities</h3><p>The final layer is metrics and entities. This is where validation is tied directly to the fields that power business-critical metrics and canonical entity definitions.</p><p>At this point, I’m thinking less about individual fields in isolation and more about how they roll up. Maybe I have a revenue metric built on top of payment and refund amounts. Or a canonical definition of a customer that depends on email domains, recent transactions, and activity. The key idea is that there’s a mapping between fields and the metrics or entities they support.</p><p>Here’s what that looks like in code:</p><pre>from datetime import datetime<br>from enum import StrEnum, auto<br>from decimal import Decimal<br>from typing import Annotated, Self<br><br>from pydantic import BaseModel, Field<br><br><br>class InvoiceStatus(StrEnum):<br>  DRAFT = auto()<br>  PENDING = auto()<br>  PAID = auto()<br>  CANCELLED = auto()<br><br><br>PostalCodeAbbreviatedState = Annotated[<br>    str,<br>    Field(pattern=&quot;^(AL|AK|AZ|...|WY)$&quot;)<br>]<br><br># Field that validates currency formats<br>Currency = Annotated[<br>    Decimal,<br>    Field(gt=0, decimal_places=2)<br>]<br><br><br>class ApiResponse(BaseModel):<br>    id: str<br>    name: str | None = None<br>    description: str | None = None<br>    state: PostalCodeAbbreviatedState<br>    payment_amount: Currency<br>    refund_amount: Currency<br>    invoice_status: InvoiceStatus<br>    created_at: datetime<br>    last_updated: datetime<br><br>    @model_validator(mode=&quot;after&quot;)<br>    def validate_paid_amount_for_status(self) -&gt; Self:<br>        if self.invoice_status in {&quot;draft&quot;, &quot;pending&quot;}:<br>            if self.paid_amount not in {None, Decimal(&quot;0.00&quot;)}:<br>                raise ValueError(<br>                    &quot;paid_amount must be 0.00 or null when invoice_status is DRAFT or PENDING&quot;<br>                )<br>        return self</pre><p>Here, payment_amount and refund_amount are no longer just numeric fields—they directly contribute to something like a revenue metric. I validate that they are properly formatted currency amounts and not negative (even though refunds can sometimes appear as negative, we’ll assume positive for this example). This helps avoid skewing revenue if the producer sends an incorrect negative value.</p><p>These are the most important checks to get right. Leadership teams tend to have their eyes glued to these numbers, and if something goes wrong, it’s usually not long before someone notices. When that happens, it can turn your day very hectic very quickly.</p><p>There are two big challenges that come to mind at this step. The first is simply keeping up with metric and entity evolution. Metrics change, definitions get refined, and new entities are introduced — all often downstream from where the data is ingested. I’m a big believer that validation should happen as early as possible — where the fields are defined — but that creates a gap. As data engineers, we need some way to stay aligned with how those fields are being used. That might mean maintaining a data dictionary, building tighter feedback loops with analysts, or even leaning on tooling or agents that understand how transformations map fields to metrics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4F4WehylQdVoktPANEFuAA.png" /></figure><p>The second challenge is less technical and more organizational. In reality, people don’t always communicate as much as they should, and decisions get made without the right stakeholders involved. A common scenario is a new metric being defined on top of a field that we can’t confidently validate because the producer never set expectations for it. Silos happen, and suddenly an unvalidated field becomes part of a critical metric. That puts us in a tough spot — we can’t introduce strict validation without a contract to back it. But all isn’t lost. Sometimes, we can still apply lightweight, behavior-based validation.</p><p>For example, if payment_amount is nullable and we observe that nulls often represent zero-value transactions, we can normalize that:</p><pre>from decimal import Decimal<br>from pydantic import field_validator<br><br>class ApiResponse(BaseModel):<br>    # ... existing fields ...<br><br>    payment_amount: PaymentAmount<br><br>    # Turn nulls into 0.00 payment amounts<br>    @field_validator(&quot;payment_amount&quot;, mode=&quot;before&quot;)<br>    @classmethod<br>    def normalize_payment_amount(cls, value):<br>        if value is None:<br>            return Decimal(&quot;0.00&quot;)<br>        return value</pre><p>For totals that use SUM() in SQL, NULL values are ignored, so the impact is minimal even without this validation. However, if we’re counting transactions, things can get tricky. Depending on how the count is written, we might skip invoices with NULL amounts and end up understating the number of transactions.</p><p>This isn’t enforcing a strict contract from the producer — it’s codifying a pattern we understand from the business. And that’s really the point: validation only works when there’s some expectation behind a field, whether it comes from the producer or emerges from consistent business behavior. Without that, validation starts to break down. If a field fails validation and no one owns its definition, who do you even go to fix it?</p><p>This layer is where the trust you’ve built with leadership is at the highest risk.</p><h3>Summary: The Four Layers of Validation</h3><ul><li><strong>Schema Validation</strong> → Establishes the baseline shape of the data (required fields, types, nullability). Driven almost entirely by the producer’s contract. Keep it simple and avoid being overly strict.</li><li><strong>Value Formats</strong> → Validates how values are structured (e.g., state abbreviations, IDs, codes). Still contract-driven, but requires judgment on what actually matters downstream.</li><li><strong>Business Rules</strong> → Ensures fields make sense <em>together</em> (e.g., invoice_status and paid_amount). This is where domain knowledge comes in and where validation starts protecting real data workflows.</li><li><strong>Metrics &amp; Entities</strong> → Focuses on fields that power critical metrics and definitions (e.g., revenue). These are the highest-impact checks — and where mistakes can directly affect trust with leadership.</li></ul><p><strong>Big takeaway:</strong> Not all validation is created equal. The goal isn’t to validate everything — it’s to apply the <em>right</em> level of validation where it actually protects downstream value.</p><p>Welp, thanks for reading! Connect with me on <a href="https://www.linkedin.com/in/jonathan-duran-80974a183/">LinkedIn</a> or <a href="https://x.com/jon__duran">X</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c6aac0f752dd" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/validate-smarter-at-the-row-level-a-four-layer-approach-c6aac0f752dd">Validate Smarter at the Row Level: A Four-Layer Approach</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes Probes 101: What You Need to Know]]></title>
            <link>https://aws.plainenglish.io/kubernetes-probes-101-what-you-need-to-know-bf2b2a8a6281?source=rss-89051b7c318f------2</link>
            <guid isPermaLink="false">https://medium.com/p/bf2b2a8a6281</guid>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Jonathan Duran]]></dc:creator>
            <pubDate>Mon, 02 Feb 2026 04:15:22 GMT</pubDate>
            <atom:updated>2026-02-03T06:18:10.982Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction to Kubernetes Probes</h3><p>Kubernetes uses probes to check if your containers are running properly. Think of them as health checks that run throughout your container’s lifecycle. The kubelet running on each node is responsible for executing these probes and acting on the results.</p><p>There are three probe types that serve different purposes: startup probes, readiness probes, and liveness probes. We’ll cover each of these in detail in the following sections.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/781/1*0rBL9PLJmNkrHONup_e2kg.png" /></figure><p>Let’s look at three common probe types you’ll work with:</p><ul><li><strong>HTTP probes:</strong> send an HTTP GET request to a specified endpoint. If the response status is between 200–399, the probe succeeds.</li><li><strong>TCP probes:</strong> attempt to open a TCP connection to a specified port. If the connection is established, the probe succeeds.</li><li><strong>Exec probes:</strong> run a command inside the container. If the command exits with status code 0, the probe succeeds.</li></ul><p>Here’s what a basic HTTP probe configuration looks like in your pod spec:</p><pre>livenessProbe:<br>  httpGet:<br>    path: /health<br>    port: 8080<br>  initialDelaySeconds: 30<br>  periodSeconds: 10<br>  failureThreshold: 3</pre><p>A few key parameters worth understanding:</p><p><strong>initialDelaySeconds</strong> , How long to wait after the container starts before running the first probe. This gives your application time to initialize.</p><p><strong>periodSeconds</strong> , How often to run the probe after the initial delay.</p><p><strong>failureThreshold</strong> , How many consecutive failures before Kubernetes takes action (the action depends on which probe type you’re using).</p><h3>Startup Probes</h3><p>Startup probes determine whether your container has successfully started. This is particularly useful for containers that have long initialization times.</p><p>When you configure a startup probe, the readiness and liveness probes remain disabled until the startup probe succeeds. This prevents Kubernetes from prematurely checking if your application is ready or healthy before it’s even finished starting up.</p><p>If the startup probe fails after exceeding the configured failure threshold, the kubelet kills the container and applies the pod’s restart policy. This means your container will be restarted if the restart policy allows it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/1*xR518Sb2wjjuIgEkRCts2g.png" /></figure><h3>Readiness Probes</h3><p>Readiness probes determine whether your container is ready to accept traffic. Just because a container is running doesn’t mean it’s prepared to handle requests.</p><p>For example, your application might be running but still loading data into memory, establishing database connections, or warming up caches. During these periods, the container shouldn’t receive traffic even though it’s technically alive.</p><p>When a readiness probe fails, Kubernetes removes the pod’s IP address from the endpoints of all services that match the pod. This means the pod stops receiving traffic from the service until the probe succeeds again. The pod itself continues running , it’s simply taken out of the load balancing rotation.</p><p>If the readiness probe continues to fail, the pod remains removed from service endpoints but isn’t restarted. The kubelet keeps checking periodically, and the pod will be added back to the service once the probe succeeds.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/733/1*WdlUu-Xx2TRR1dAhFNZMDw.png" /></figure><h3>Liveness Probes</h3><p>Liveness probes determine whether your container is still running properly. They help Kubernetes detect when an application has entered a broken state from which it can’t recover on its own.</p><p>For example, your application might hit a deadlock, enter an infinite loop, or encounter an unrecoverable error that leaves the process running but unable to make progress. In these situations, the container is technically alive from the OS perspective, but functionally dead from an application perspective.</p><p>The key difference between liveness and readiness probes is their purpose and action. Readiness probes control traffic flow , they remove pods from service endpoints when they’re temporarily unable to serve requests. Liveness probes control container lifecycle , they restart containers that are broken beyond self-recovery.</p><p>When a liveness probe fails after exceeding the failure threshold, the kubelet kills the container and restarts it according to the pod’s restart policy. This gives your application a fresh start and a chance to recover from the broken state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oKjj5c5ufCJ8vU7immExFA.png" /></figure><h4>Thanks for reading! Feel free to reach out and say hi 👋</h4><ul><li><a href="https://www.linkedin.com/in/jonathan-duran-80974a183/">LinkedIn</a></li><li><a href="https://x.com/jon__duran">X</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bf2b2a8a6281" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/kubernetes-probes-101-what-you-need-to-know-bf2b2a8a6281">Kubernetes Probes 101: What You Need to Know</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[PostgreSQL WAL Internals for Data Engineers]]></title>
            <link>https://blog.dataengineerthings.org/postgresql-wal-internals-for-data-engineers-ef6229584a99?source=rss-89051b7c318f------2</link>
            <guid isPermaLink="false">https://medium.com/p/ef6229584a99</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Jonathan Duran]]></dc:creator>
            <pubDate>Tue, 28 Oct 2025 00:54:02 GMT</pubDate>
            <atom:updated>2025-10-30T17:40:35.699Z</atom:updated>
            <content:encoded><![CDATA[<h3>Understanding Postgres WAL</h3><p>I was in the middle of a CDC project recently and realized I’d never really taken the time to dig into the Postgres internals that make streaming transactions possible. We all know Change Data Capture as the magic that lets us push database changes in near real time to systems like Kafka, Snowflake, or a data lake — but under the hood, there’s a specific piece of Postgres engineering doing the heavy lifting: the Write-Ahead Log (WAL).</p><p>This article focuses on that component — what the WAL is, how it underpins Postgres’s reliability, and the role it plays in enabling CDC. We’ll walk through how Postgres handles transactions from memory to disk, how it avoids expensive I/O, and how it recovers gracefully after a crash.</p><h3>The Transaction Lifecycle and WAL</h3><p>Every transaction in Postgres follows a predictable sequence:</p><ol><li>You issue a query — say an INSERT or UPDATE.</li><li>Postgres locates the relevant data page (an 8 KB chunk of table or index data).</li><li>The change is made <em>in memory</em>, not directly on disk.</li><li>A record of that change is written to the Write-Ahead Log (WAL).</li><li>On COMMIT, the WAL is flushed to disk, guaranteeing durability.</li><li>Later, Postgres writes the modified (“dirty”) data pages to disk in bulk, during a process called a checkpoint.</li></ol><p>That ordering — write to WAL first, then data later — is the “write-ahead” part of <em>Write-Ahead Log</em>.</p><p>It’s what guarantees that committed transactions survive crashes, even if the actual table data hasn’t been updated yet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/1*iH8cEboKgvapinln2-2DBw.png" /><figcaption>Lifecycle of a transaction</figcaption></figure><h3>Shared Memory, Buffers, and How Updates Happen</h3><p>When Postgres runs, it allocates two main memory areas:</p><pre>+--------------------+----------------+----------------------------------------+<br>|    Memory Area     |    Parameter   |               Purpose                  | <br>+--------------------+----------------+----------------------------------------+<br>| Shared Buffer Pool | shared_buffers | Sets the amount of memory the database |<br>|                    |                | server uses for shared memory buffers. |<br>+--------------------+----------------+----------------------------------------+<br>|     WAL Buffer     |  wal_buffers   | The amount of shared memory used for   |<br>|                    |                | WAL data that has not yet been written |<br>|                    |                | to disk                                |<br>+--------------------+----------------+----------------------------------------+</pre><p>Here’s what happens when you modify data:</p><ol><li>Fetching a Page — If the target page isn’t already in memory, Postgres reads it from disk into the shared buffer pool. This is direct: each table page has a known byte offset on disk, so Postgres doesn’t “search” — it just reads the specific 8 KB block it needs.</li><li>Modifying in Memory — Postgres updates the in-memory copy of that page. The page is now <em>dirty</em> — meaning it contains changes not yet written to disk.</li><li>Creating a WAL Record — The backend generates a WAL record describing <em>what changed</em> and appends it to the WAL buffer in shared memory.</li></ol><p>At this point, the transaction’s data is in memory, and the change is recorded in WAL memory — but nothing has yet been persisted to disk.</p><h3>WAL Flush vs. Data Flush</h3><p>Here’s where Postgres’s design really shines.</p><p>Flushing the WAL and flushing the data files happen at very different times, for very different reasons.</p><h4>WAL Flush (at COMMIT)</h4><p>When you commit a transaction, Postgres calls XLogFlush(), which ensures all relevant WAL records are safely written to disk. Those records are first created in memory by XLogInsertRecord(), the function responsible for assembling and appending each change description into the shared WAL buffer before it’s flushed. In other words, XLogInsertRecord() logs the change, and XLogFlush() makes it durable.</p><p>WAL files live in the pg_wal/ directory as a sequence of 16 MB segment files, each filled sequentially before rolling over to the next. Older segments are either recycled or removed once they’re no longer needed for recovery or replication. This simple, append-only structure keeps WAL writes fast and efficient.</p><p>Parameters that control this behavior:</p><ul><li>wal_sync_method – how Postgres asks the OS to flush to disk (fsync, fdatasync, etc.).</li><li>commit_delay and commit_siblings – allow group commit, batching multiple commits into one flush for throughput.</li><li>synchronous_commit – controls whether Postgres waits for the WAL flush before returning success (default: on).</li></ul><p>Because WAL writes are sequential, they’re cheap. That’s why it’s not a problem to flush the WAL to disk on every commit — it’s a single append, not a random write scattered across the disk.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/769/1*e6Volg0hB2PtMTHHKb06AA.png" /><figcaption>WAL entries created in memory and flushed to disk.</figcaption></figure><h4>Data Flush (Checkpoints)</h4><p>The data itself, living in the shared buffer pool, doesn’t get written out on every transaction.</p><p>Instead, Postgres defers those writes to a checkpoint, which happens periodically or when too much WAL accumulates.</p><p>At checkpoint time:</p><ul><li>All dirty pages in shared buffers are written to disk.</li><li>A checkpoint record is added to the WAL.</li><li>Older WAL files that are no longer needed for recovery can be recycled or removed.</li></ul><p>This design significantly reduces random I/O. Instead of writing data constantly in random patterns, Postgres batches and spreads out the work, giving the OS time to optimize writes.</p><p>Parameters that tune this behavior:</p><ul><li>checkpoint_timeout – maximum time between checkpoints (default: 5 min).</li><li>max_wal_size – how much WAL can accumulate before a checkpoint is triggered.</li><li>checkpoint_completion_target – how smoothly to spread out checkpoint I/O (default: 0.9).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/753/1*VNDn-cU31MSh5qBFsr3mvw.png" /><figcaption>Time-based Checkpoints</figcaption></figure><h3>Why It Works: Sequential WAL, Deferred Data</h3><p>It’s worth emphasizing just how clever this is.</p><ul><li>WAL writes are fast because they’re <em>sequential</em> and small (a few hundred bytes per record).</li><li>Data writes are slow because they’re <em>random</em> and big (8 KB pages scattered across files).</li></ul><p>By writing to the WAL first, Postgres guarantees durability without paying the random write penalty.</p><p>Then, by flushing data lazily during checkpoints, it minimizes I/O overhead during normal operation.</p><p>This separation of responsibilities — WAL for reliability, data writes for performance — is what lets Postgres scale to huge transaction volumes without destroying disk throughput.</p><h3>What Happens During a Checkpoint</h3><p>A checkpoint marks a point in time when Postgres says, “everything before here is definitely on disk.”</p><p>Here’s what happens under the hood:</p><ol><li>The checkpointer process starts flushing dirty buffers to disk.</li><li>When all dirty pages are written, it writes a checkpoint record to the WAL.</li><li>That record contains the <em>redo point</em> — the WAL position where crash recovery should start if needed.</li><li>WAL segments older than the redo point can now be recycled or removed.</li></ol><p>After a checkpoint, new changes begin again: any page modified for the first time after that checkpoint triggers a full page image write to the WAL, to protect against torn pages.</p><h3>Crash Recovery: Replaying the WAL</h3><p>Now imagine the server crashes mid-transaction.</p><p>What happens when Postgres restarts?</p><ol><li>It reads the latest checkpoint record in the WAL.</li><li>The checkpoint’s redo point tells it where to start.</li><li>It replays all WAL records from that point forward, applying each change in order.</li><li>If any pages were partially written (torn), Postgres restores them using full-page images stored in the WAL.</li><li>Checksums verify the data’s integrity before applying each record.</li></ol><p>Once it reaches the end of the WAL, the database is guaranteed to be in a consistent state — every committed transaction is present, and every uncommitted one is rolled back.</p><p>That’s why the WAL isn’t just a log — it’s the single source of truth for recovery. As long as the WAL is intact, the database can rebuild itself.</p><h3>Why This Matters for CDC and Replication</h3><p>If you’re working with CDC or logical replication, this same WAL is where the magic happens.</p><p>Postgres’s logical decoding process reads changes directly from WAL records, translating them into row-level events.</p><p>Replication slots, publications, and decoding plugins (like pgoutput) all depend on this log being accurate and complete.</p><p>That means every bit of reliability we just discussed — WAL flushes, checkpoints, full-page writes — also protects the correctness of the CDC stream.</p><p>When you understand WAL, you’re not just understanding crash recovery — you’re understanding the backbone of Postgres replication itself.</p><h3>Wrapping Up</h3><ul><li>Every change in Postgres flows through the WAL, ensuring durability before touching data files.</li><li>The shared buffer pool caches data pages; the WAL buffer caches change records.</li><li>WAL flushes are fast (sequential); data writes are deferred to checkpoints (random but batched).</li><li>Checkpoints clean up and mark safe recovery points.</li><li>If Postgres crashes, WAL replay rebuilds the database exactly as it was.</li></ul><p>The next time you see CDC events flowing smoothly from Postgres to your downstream systems, remember — it’s the WAL behind the scenes making it possible, steadily recording each change as it happens.</p><p>As always, feel free to follow me on <a href="https://x.com/jon__duran">X</a> or connect on <a href="https://www.linkedin.com/in/jonathan-duran-80974a183/">LinkedIn</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ef6229584a99" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/postgresql-wal-internals-for-data-engineers-ef6229584a99">PostgreSQL WAL Internals for Data Engineers</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Side Project Helpers: Automating Postgres Inserts, Updates, and Deletes]]></title>
            <link>https://medium.com/towards-data-engineering/side-project-helpers-automating-postgres-inserts-updates-and-deletes-c96d6a5073dc?source=rss-89051b7c318f------2</link>
            <guid isPermaLink="false">https://medium.com/p/c96d6a5073dc</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Jonathan Duran]]></dc:creator>
            <pubDate>Sun, 07 Sep 2025 17:41:56 GMT</pubDate>
            <atom:updated>2025-09-09T15:58:55.549Z</atom:updated>
            <content:encoded><![CDATA[<h3>Side-Projects: Automating Postgres Inserts, Updates, and Deletes</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6t1bmDaXsbMN441w5vsisA.png" /><figcaption>Artist: ChatGippity</figcaption></figure><p>As a data engineer, I’m always tinkering with side projects. Over the years, that’s meant building lots of little helper scripts — small tools to make my life easier when I’m learning something new or testing out an idea. Recently, I took a step back and realized <strong><em>I’ve got a whole toolbox of these utilities lying around</em></strong>. Instead of keeping them to myself, I thought: why not share them with the world?</p><p>I’m kicking off this little initiative with one of my favorites: a simple application that makes it easy to load and modify a Postgres database with data. It creates a fake orders table and lets you insert, update, or delete as many rows as you want.</p><p>Why does this matter? If you’ve ever tried to learn or test <strong>Change Data Capture (CDC)</strong> pipelines, you know the pain of needing a “real-enough” source system that you can mess with. This app has been my go-to for spinning up quick, disposable datasets so I can focus on CDC logic instead of hand-crafting SQL statements every time.</p><h3>What It Does</h3><p>Here’s the gist:</p><ul><li>Creates an orders table in Postgres (if it doesn’t already exist).</li><li>Inserts fake rows with realistic data: UUIDs, statuses (pending, paid, shipped, etc.), amounts, and timestamps.</li><li>Updates existing rows by changing their status and updating the timestamp.</li><li>Deletes rows (without crashing if you ask for more rows than exist).</li><li>All of this is wrapped in a friendly CLI interface so you don’t need to crack open SQL clients to get going.</li></ul><h3>Running It</h3><h4>Prerequisites</h4><p>Before you get started, make sure you have:</p><ul><li>Git installed so you can clone the repo.</li><li>Docker and Docker Compose installed (for running a local Postgres instance).</li><li>Python 3.12+ installed locally.</li><li><a href="https://docs.astral.sh/uv/getting-started/installation/">uv</a> for installing and running the app.</li></ul><p>First, clone the project and move into the directory:</p><pre>git clone https://github.com/jduran9987/tooling-postgres-cdc.git<br>cd tooling-postgres-cdc</pre><h4>Start Postgres Locally</h4><p>This project comes with a ready-to-go docker-compose.yaml file. Spinning up Postgres is as simple as:</p><pre># Start Postgres in the background<br>docker compose up -d</pre><p>This will:</p><ul><li>Launch a Postgres container on port <strong>5432</strong></li><li>Use the credentials from your .env file (username: postgres, password: postgres by default)</li><li>Persist data in a Docker-managed volume called postgres_data</li></ul><p>To stop Postgres:</p><pre># Stop Postgres containers<br>docker compose down<br><br># Or, if you also want to remove all data (start fresh)<br>docker compose down -v</pre><h4>Pointing to a Remote Postgres</h4><p>If you’d rather use another Postgres instance (say, on a cloud provider or your own machine), just edit the .env file:</p><pre>POSTGRES_DB=mydb<br>POSTGRES_USER=myuser<br>POSTGRES_PASSWORD=supersecret<br>POSTGRES_HOST=some.remote.host<br>POSTGRES_PORT=5432</pre><p>The app will connect to whatever values you supply here.</p><h4>Run the Application</h4><p>Once Postgres is up and your .env file is set, you can start issuing commands:</p><pre># Insert 10 rows<br>uv run python src/main.py --action insert --num-rows 10<br><br># Update 5 rows<br>uv run python src/main.py --action update --num-rows 5<br><br># Delete 3 rows<br>uv run python src/main.py --action delete --num-rows 3<br><br># Drop the table entirely<br>uv run python src/main.py --clean</pre><h4>Verifying Data Inside the Local Container</h4><p>Once you’ve inserted, updated, or deleted rows with the app, you might want to peek inside the Postgres container to confirm the data. You can do this with docker exec and the built-in psql client.</p><pre># Open a psql shell inside the Postgres container<br>docker exec -it postgres_db psql -U postgres -d postgres</pre><p>From there, you can run:</p><pre># Using the PSQL client in the container<br>\dt;  # &lt;--- Lists the tables in the database.<br>\d orders;  # &lt;--- Provides information about the orders table.<br>SELECT * FROM orders LIMIT 5;  # &lt;--- Queries the orders table.</pre><h4>Example Output</h4><p>Here’s a sample of what you might see after running --action insert --num-rows 5:</p><pre>           id                         |  status   | total_amount_cents |  created_at   | last_updated_at<br>--------------------------------------+-----------+--------------------+---------------+-----------------<br> 4e8a8e9c-fbb0-4a93-9c2b-7b1d3d8e9af5 | pending   |               4821 | 1712435805123 | 1712435805123<br> 09cf0c89-3c2a-4a2f-bcc2-2a2c2e5efbd0 | paid      |               9021 | 1712435806123 | 1712435806123<br> 3b4a8a2e-d23c-4590-b14e-2c23c5b23d90 | shipped   |               3011 | 1712435807123 | 1712435807123<br> 72b6ff0b-0a21-46f2-92a9-8c65bb123456 | cancelled |               7452 | 1712435808123 | 1712435808123<br> e2f7c1d4-13bc-44df-a9f8-72e6ff7890aa | refunded  |               1500 | 1712435809123 | 1712435809123</pre><ul><li>id → a random UUID (unique per row).</li><li>status → one of: pending, paid, shipped, cancelled, refunded.</li><li>total_amount_cents → random integer between 1000 and 10000 (represents cents).</li><li>created_at and last_updated_at → epoch timestamps in milliseconds.</li></ul><p>If you run an <strong>update</strong>, you’ll see the status and last_updated_at fields change, while id, total_amount_cents, and created_at stay the same.</p><p>If you run a <strong>delete</strong>, the number of rows will shrink accordingly.</p><h3>Orders Table Schema</h3><p>The application manages a single table called <strong>orders</strong>.</p><pre>| Column               | Type    | Description                                                           | Changes on Update? |<br>| -------------------- | ------- | --------------------------------------------------------------------- | ------------------ |<br>| `id`                 | TEXT    | Primary key, unique identifier (UUID).                                | ❌ No               |<br>| `status`             | TEXT    | Order status (`pending`, `paid`, `shipped`, `cancelled`, `refunded`). | ✅ Yes              |<br>| `total_amount_cents` | INTEGER | Order amount in cents.                                                | ❌ No               |<br>| `created_at`         | BIGINT  | Epoch timestamp (ms) when the row was created.                        | ❌ No               |<br>| `last_updated_at`    | BIGINT  | Epoch timestamp (ms) when the row was last updated.                   | ✅ Yes              |</pre><p>with SQL Definition:</p><pre>CREATE TABLE IF NOT EXISTS orders (<br>    id                 TEXT    PRIMARY KEY,<br>    status             TEXT    NOT NULL,<br>    total_amount_cents INTEGER NOT NULL,<br>    created_at         BIGINT  NOT NULL,<br>    last_updated_at    BIGINT  NOT NULL<br>);</pre><h3>Project Ideas</h3><p>Once you have this application set up and generating changes in the orders table, you can extend it into bigger data engineering workflows.</p><p>A great next step is to configure <strong>Debezium</strong> with <strong>Kafka</strong> to capture the insert, update, and delete events happening in Postgres, then stream those changes into a data lake like <strong>Amazon S3</strong>.</p><p>This lets you experiment with a real <strong>Change Data Capture (CDC)</strong> pipeline end-to-end:</p><ul><li><strong>Postgres</strong> → <strong>Debezium</strong> → <strong>Kafka</strong> → <strong>S3</strong></li></ul><p>From there, you can practice building transformations, downstream consumers, or even data lakehouse patterns on top of the captured change events.</p><p>Thanks for taking the time to check out this project!</p><p>I hope it gives you a simple, practical way to spin up fake Postgres data and experiment with concepts like CDC. If you end up using it in your own projects — or if you have ideas to make it better — I’d love to hear about it.</p><p>Feel free to reach out on <a href="https://x.com/jon__duran">X</a> or <a href="https://www.linkedin.com/in/jonathan-duran-80974a183/">LinkedIn</a> to connect and share your thoughts.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c96d6a5073dc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-data-engineering/side-project-helpers-automating-postgres-inserts-updates-and-deletes-c96d6a5073dc">Side Project Helpers: Automating Postgres Inserts, Updates, and Deletes</a> was originally published in <a href="https://medium.com/towards-data-engineering">Towards Data Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Waiter Will Be Right With You (Request–Response Explained)]]></title>
            <link>https://python.plainenglish.io/the-waiter-will-be-right-with-you-request-response-explained-5ef3920f9c7b?source=rss-89051b7c318f------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ef3920f9c7b</guid>
            <category><![CDATA[api]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-architecture]]></category>
            <dc:creator><![CDATA[Jonathan Duran]]></dc:creator>
            <pubDate>Sun, 24 Aug 2025 05:06:51 GMT</pubDate>
            <atom:updated>2025-08-25T00:34:33.956Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UtD64YfSF3FZrd-8tlrcaw.png" /></figure><p>The request–response pattern — it’s the typical way services communicate in distributed systems. One microservice (the client) sends a request to another service (the server), and that service returns a response.</p><p>Let’s quickly walk through a few core traits of request–response — the pieces that shape how it behaves in real systems.</p><h4>Synchronous</h4><p>First thing to know: this pattern is synchronous. The client’s business logic waits — blocks — until a response comes back, keeping the connection open the whole time. In other words, the caller pauses until it has an answer (success or error) to act on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Uo9c-EmA2lObteQsywZIzw.png" /></figure><h4>Strong coupling in space and time</h4><p>Unlike asynchronous, buffered approaches (queues, pub/sub), request–response expects both sides to be online at the same time so the data exchange can complete. If the server is offline or unreachable, the client can’t finish its work yet — it usually needs to retry the request before it can proceed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f34uVwgLqhOWQkrACm9Cyg.png" /></figure><p>On top of this <strong>time coupling</strong>, there’s also <strong>space coupling</strong>. The client has to know exactly where the server lives and the details of its API: the URL, endpoints, request and response formats, and error codes. This creates a tight dependency — if the server moves, changes an endpoint, or alters its contract, the client may break unless both evolve together.</p><h4>Data reusability</h4><p>Another angle to think about is data reuse. In a request–response world, powering something like a data warehouse or data lake usually means reaching straight into a service’s database. That might look like nightly ETL jobs or change data capture (CDC) pipelines pulling records out and landing them somewhere central for analytics. It works, but it often feels bolted on — the service wasn’t built to share data this way, so you end up tapping the database directly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nSCKZGBOqj_AuUIp70tE3g.png" /></figure><p>Contrast that with an event-driven setup, where services publish events as they happen and those events can stream straight into an analytics platform. That makes the same data available immediately for machine learning, reporting, or even downstream data apps, without having to extract it after the fact (you still have the option of running batch ETL jobs in event-driven architectures).</p><h4>Consistency</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gTlWk1c24peet3zMII9YGQ.png" /></figure><p>One bright side of request–response is consistency. Because services are talking to each other directly — publishing updates and waiting for a reply — the data they exchange is typically consistent right away. That’s different from many asynchronous patterns, where events flow through queues or streams and consumers might not process them until later. In those cases, you accept eventual consistency across systems. With request–response, the client usually knows immediately whether the update succeeded, and both sides can move forward on the same page.</p><h3>Let’s see some code!</h3><p>To make this a bit more concrete, let’s step into some code and see what the request–response pattern looks like in practice. We’ll keep it simple with two services working together:</p><ul><li><strong>User Service</strong> (client): exposes POST /users. It validates input, creates the user, and—<em>as part of the same request</em>—calls the Notification Service to send a welcome email.</li><li><strong>Notification Service</strong> (server): exposes POST /notifications. It accepts a JSON payload (user id, email, message, timestamp) and responds with a simple success or an error.</li></ul><p>The flow is straightforward:</p><ol><li>Client hits <strong>User Service</strong> → POST /users.</li><li>After the user is persisted, the User Service makes an HTTP call to <strong>Notification Service</strong> → POST /notifications.</li><li>The <strong>User Service</strong> <strong>awaits</strong> that response before it returns to the original caller.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/675/1*w0XNsL7neFkr5-pLVEp1sg.png" /></figure><h4>User Service</h4><pre>import asyncio<br>import httpx<br>import os<br>import time<br><br>from fastapi import FastAPI, HTTPException<br>from pydantic import BaseModel<br><br><br>app = FastAPI(title=&quot;User Service&quot;)<br><br># Application variables<br>NOTIFICATION_SERVICE_BASE_URL = os.environ[&quot;NOTIFICATION_SERVICE_BASE_URL&quot;]<br><br># Retry configuration<br>RETRY_ATTEMPTS = 3 # total tries<br>RETRY_SLEEP = 0.5 # seconds between tries<br><br># Simple in-memory storage<br>users = []<br><br><br>class UserCreate(BaseModel):<br>    name: str<br>    email: str<br><br><br>class UserResponse(BaseModel):<br>    id: int<br>    name: str<br>    email: str<br><br><br>@app.get(&quot;/&quot;)<br>async def root() -&gt; dict[str, str]:<br>    return {&quot;service&quot;: &quot;User Service&quot;, &quot;status&quot;: &quot;running&quot;}<br><br><br>@app.post(&quot;/users&quot;, response_model=UserResponse)<br>async def create_user(user: UserCreate) -&gt; UserResponse:<br>    id = len(users) + 1<br>    <br>    user_data = {<br>        &quot;id&quot;: id,<br>        &quot;name&quot;: user.name,<br>        &quot;email&quot;: user.email<br>    }<br>    users.append(user_data)<br><br>    async with httpx.AsyncClient(timeout=5.0) as client:<br>        for attempt in range(RETRY_ATTEMPTS):<br>            payload = {<br>                &quot;user_id&quot;: id,<br>                &quot;message&quot;: f&quot;Welcome, {user.name}!&quot;,<br>                &quot;email&quot;: user.email,<br>                &quot;timestamp&quot;: time.time()<br>            }<br>            response = await client.post(f&quot;{NOTIFICATION_SERVICE_BASE_URL}/notifications&quot;, json=payload)<br>            <br>            # Only retry on explicit 500 from Notification Service<br>            if response.status_code == 500:<br>                if attempt == RETRY_ATTEMPTS - 1:<br>                    raise HTTPException(<br>                        status_code=500,<br>                        detail=&quot;Failed to get a successful response from Notification Service.&quot;<br>                    )<br>                await asyncio.sleep(RETRY_SLEEP)<br><br>            # Any other non-2xx should raise (no retry here)<br>            response.raise_for_status()<br><br>    return UserResponse(**user_data)</pre><h4>Notifications Service</h4><pre>from fastapi import FastAPI<br>from pydantic import BaseModel<br><br><br>app = FastAPI(title=&quot;Notification Service&quot;)<br><br># Simple in-memory storage<br>notifications = []<br><br><br>class NotificationCreate(BaseModel):<br>    user_id: int<br>    message: str<br>    email: str<br>    timestamp: float<br><br><br>class NotificationResponse(BaseModel):<br>    id: int<br>    user_id: int<br>    message: str<br>    email: str<br>    timestamp: float<br><br><br>@app.get(&quot;/&quot;)<br>async def root() -&gt; dict[str, str]:<br>    return {&quot;service&quot;: &quot;Notification Service&quot;, &quot;status&quot;: &quot;running&quot;}<br><br><br>@app.get(&quot;/notifications&quot;, response_model=list[NotificationResponse])<br>async def get_notifications() -&gt; list[NotificationResponse]:<br>    return [NotificationResponse(**notification) for notification in notifications]<br><br><br>@app.post(&quot;/notifications&quot;, response_model=NotificationResponse)<br>async def create_notification(notification: NotificationCreate) -&gt; NotificationResponse:<br>    &quot;&quot;&quot;<br>    For simplicity, this function only &quot;persists&quot; the data it receives<br>    from the User Service. However, we can imagine that it also calls<br>    some external system to send that user a welcome email.<br>    &quot;&quot;&quot;<br>    id = len(notifications) + 1<br><br>    notification_data = {<br>        &quot;id&quot;: id,<br>        &quot;user_id&quot;: notification.user_id,<br>        &quot;message&quot;: notification.message,<br>        &quot;email&quot;: notification.email,<br>        &quot;timestamp&quot;: notification.timestamp<br>    }<br><br>    notifications.append(notification_data)<br><br>    return NotificationResponse(**notification_data)</pre><p>It’s not important to understand every line of code here — what matters is seeing how the concepts we introduced earlier show up in practice. And since our Notification Service doesn’t actually send emails, we’ll have to use our imagination a bit that it happens magically behind the scenes.</p><p>Let’s highlight a few things in this microservice code that tie back to some of the core traits mentioned in the theory section of this article:</p><ul><li><strong>Tight coupling in space</strong> — The User Service depends on knowing exactly where the Notification Service lives. It expects the NOTIFICATION_SERVICE_BASE_URL environment variable to be set and also assumes the /notifications path exists.</li><li><strong>Tight coupling in time</strong> — The retry and timeout logic around the call shows what happens when the server isn’t immediately available. The client must wait and retry before it can finish its work.</li><li><strong>API details</strong> — The User Service needs to know exactly what data shape the Notification Service expects (user_id, message, email, timestamp). That’s the shared contract that couples them.</li><li><strong>Synchronous flow</strong> — Notice how the create_user route is blocked until the Notification Service responds. The rest of the business logic doesn’t continue until the request–response exchange finishes.</li></ul><h3>Wrapping up</h3><p>The request–response pattern is the most typical way microservices talk to each other: one service calls, the other answers, and the caller blocks until it hears back. It’s simple, direct, and predictable — which is why it shows up everywhere. But as we’ve seen, it also brings trade-offs: services are tightly coupled in both time and space, the client has to understand the server’s API details, and availability issues ripple directly into the caller.</p><p>None of these are deal-breakers — it’s just the nature of synchronous communication. For some flows, especially user-facing ones where you need an immediate confirmation, request–response could be the best fit. For others, asynchronous or event-driven patterns may give you more flexibility.</p><p>Thanks for reading — I hope this hands-on walkthrough gave you a practical feel for how the pattern works in code and how it ties back to the bigger ideas.</p><p>As always, feel free to connect with me on <a href="https://www.linkedin.com/in/jonathan-duran-80974a183/">LinkedIn</a> and/or <a href="https://x.com/jon__duran">X</a>.</p><p>Peace ✌️</p><h3>A message from our Founder</h3><p><strong>Hey, </strong><a href="https://linkedin.com/in/sunilsandhu"><strong>Sunil</strong></a><strong> here.</strong> I wanted to take a moment to thank you for reading until the end and for being a part of this community.</p><p>Did you know that our team run these publications as a volunteer effort to over 3.5m monthly readers? <strong>We don’t receive any funding, we do this to support the community. ❤️</strong></p><p>If you want to show some love, please take a moment to <strong>follow me on </strong><a href="https://linkedin.com/in/sunilsandhu"><strong>LinkedIn</strong></a><strong>, </strong><a href="https://tiktok.com/@messyfounder"><strong>TikTok</strong></a>, <a href="https://instagram.com/sunilsandhu"><strong>Instagram</strong></a>. You can also subscribe to our <a href="https://newsletter.plainenglish.io/"><strong>weekly newsletter</strong></a>.</p><p>And before you go, don’t forget to <strong>clap</strong> and <strong>follow</strong> the writer️!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ef3920f9c7b" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/the-waiter-will-be-right-with-you-request-response-explained-5ef3920f9c7b">The Waiter Will Be Right With You (Request–Response Explained)</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Somewhat Complicated Process Of Rotating Secrets]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://aws.plainenglish.io/the-somewhat-complicated-process-of-rotating-secrets-91f0b6962336?source=rss-89051b7c318f------2"><img src="https://cdn-images-1.medium.com/max/1536/1*k6M5qube5Pw0f354lG7GHw.png" width="1536"></a></p><p class="medium-feed-snippet">A deep dive on the four steps required to rotate a secret.</p><p class="medium-feed-link"><a href="https://aws.plainenglish.io/the-somewhat-complicated-process-of-rotating-secrets-91f0b6962336?source=rss-89051b7c318f------2">Continue reading on AWS in Plain English »</a></p></div>]]></description>
            <link>https://aws.plainenglish.io/the-somewhat-complicated-process-of-rotating-secrets-91f0b6962336?source=rss-89051b7c318f------2</link>
            <guid isPermaLink="false">https://medium.com/p/91f0b6962336</guid>
            <dc:creator><![CDATA[Jonathan Duran]]></dc:creator>
            <pubDate>Sun, 17 Aug 2025 05:03:28 GMT</pubDate>
            <atom:updated>2025-08-18T03:46:41.011Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Desk Drama and ClusterIPs — Kubernetes Networking as an Office Sitcom]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://aws.plainenglish.io/desk-drama-and-clusterips-kubernetes-networking-as-an-office-sitcom-dd926631568d?source=rss-89051b7c318f------2"><img src="https://cdn-images-1.medium.com/max/1536/1*V5hpB-W1TQQ1LhtzKxNWKA.png" width="1536"></a></p><p class="medium-feed-snippet">ClusterIPs&#x200A;&#x2014;&#x200A;but with offices?</p><p class="medium-feed-link"><a href="https://aws.plainenglish.io/desk-drama-and-clusterips-kubernetes-networking-as-an-office-sitcom-dd926631568d?source=rss-89051b7c318f------2">Continue reading on AWS in Plain English »</a></p></div>]]></description>
            <link>https://aws.plainenglish.io/desk-drama-and-clusterips-kubernetes-networking-as-an-office-sitcom-dd926631568d?source=rss-89051b7c318f------2</link>
            <guid isPermaLink="false">https://medium.com/p/dd926631568d</guid>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Jonathan Duran]]></dc:creator>
            <pubDate>Sun, 10 Aug 2025 23:34:33 GMT</pubDate>
            <atom:updated>2025-08-11T00:08:52.750Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Kubernetes Labels and Selectors Explained by Someone Still Figuring It Out]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://aws.plainenglish.io/kubernetes-labels-and-selectors-explained-by-someone-still-figuring-it-out-e22619dc4107?source=rss-89051b7c318f------2"><img src="https://cdn-images-1.medium.com/max/1536/1*UlOqQn1CGRcMEdUpCbuNjA.png" width="1536"></a></p><p class="medium-feed-snippet">When I first started working with Kubernetes, labels felt like one of those &#x201C;you&#x2019;ll get it once you use it&#x201D; concepts. Except&#x2026; I didn&#x2019;t get&#x2026;</p><p class="medium-feed-link"><a href="https://aws.plainenglish.io/kubernetes-labels-and-selectors-explained-by-someone-still-figuring-it-out-e22619dc4107?source=rss-89051b7c318f------2">Continue reading on AWS in Plain English »</a></p></div>]]></description>
            <link>https://aws.plainenglish.io/kubernetes-labels-and-selectors-explained-by-someone-still-figuring-it-out-e22619dc4107?source=rss-89051b7c318f------2</link>
            <guid isPermaLink="false">https://medium.com/p/e22619dc4107</guid>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Jonathan Duran]]></dc:creator>
            <pubDate>Sat, 09 Aug 2025 18:12:03 GMT</pubDate>
            <atom:updated>2025-08-10T13:59:20.979Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[DAGs That Trigger DAGs]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://blog.dataengineerthings.org/dags-that-trigger-dags-550a1b61d98a?source=rss-89051b7c318f------2"><img src="https://cdn-images-1.medium.com/max/2330/1*aAhTi-sl-DAjV66JP8oJyA.png" width="2330"></a></p><p class="medium-feed-snippet">Create dependencies between your DAGs</p><p class="medium-feed-link"><a href="https://blog.dataengineerthings.org/dags-that-trigger-dags-550a1b61d98a?source=rss-89051b7c318f------2">Continue reading on Data Engineer Things »</a></p></div>]]></description>
            <link>https://blog.dataengineerthings.org/dags-that-trigger-dags-550a1b61d98a?source=rss-89051b7c318f------2</link>
            <guid isPermaLink="false">https://medium.com/p/550a1b61d98a</guid>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Jonathan Duran]]></dc:creator>
            <pubDate>Mon, 23 Dec 2024 03:01:16 GMT</pubDate>
            <atom:updated>2024-12-23T16:41:26.698Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Notes — User Management In Ubuntu]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://jon-duran.medium.com/notes-user-management-in-ubuntu-2e7e943c9c34?source=rss-89051b7c318f------2"><img src="https://cdn-images-1.medium.com/max/2600/1*pTr8RaYTXSriTxqyRGMUDA.jpeg" width="5184"></a></p><p class="medium-feed-snippet">Follow along with Docker</p><p class="medium-feed-link"><a href="https://jon-duran.medium.com/notes-user-management-in-ubuntu-2e7e943c9c34?source=rss-89051b7c318f------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://jon-duran.medium.com/notes-user-management-in-ubuntu-2e7e943c9c34?source=rss-89051b7c318f------2</link>
            <guid isPermaLink="false">https://medium.com/p/2e7e943c9c34</guid>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[ubuntu]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Jonathan Duran]]></dc:creator>
            <pubDate>Sun, 08 Dec 2024 05:57:08 GMT</pubDate>
            <atom:updated>2024-12-08T05:57:08.745Z</atom:updated>
        </item>
    </channel>
</rss>