<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Data Intensive Dreamer on Medium]]></title>
        <description><![CDATA[Stories by Data Intensive Dreamer on Medium]]></description>
        <link>https://medium.com/@dataintensivedreamer?source=rss-6c179cb63f57------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*eghbAO8JvfGSwltpMmH32g.png</url>
            <title>Stories by Data Intensive Dreamer on Medium</title>
            <link>https://medium.com/@dataintensivedreamer?source=rss-6c179cb63f57------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 17:38:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dataintensivedreamer/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A Good First Codex Skill: The Weekly Project Update]]></title>
            <link>https://dataintensivedreamer.medium.com/a-good-first-codex-skill-the-weekly-project-update-f48a6bfcd7fd?source=rss-6c179cb63f57------2</link>
            <guid isPermaLink="false">https://medium.com/p/f48a6bfcd7fd</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Data Intensive Dreamer]]></dc:creator>
            <pubDate>Mon, 04 May 2026 06:24:32 GMT</pubDate>
            <atom:updated>2026-05-04T06:24:32.578Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Jira tells you the plan. Git tells you what actually changed.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5fVfomCQK67aiyArNIGwWg.png" /></figure><p>Weekly project updates are small documents with a surprisingly annoying cost. The writing is rarely the hard part. The real work is checking the same sources again and again: what shipped, what is still open, what changed, what needs attention next.</p><p>That is exactly the kind of repetitive work that should be automated, as long as the automation respects the way real projects work. In a software team, evidence is spread across different systems:</p><ul><li>Jira has the delivery state.</li><li>Confluence has decisions and assumptions.</li><li>Git and Bitbucket have the code trail: Git records the history, Bitbucket hosts the repository and, when needed, the pull request conversation.</li></ul><p>Those sources answer different questions. So the useful question is not whether Codex can write a nice update. The useful question is whether Codex can follow the same evidence rules every week.</p><p>For this demo, I used one simple rule: work is shipped only when Jira and Git agree.</p><p>That means:</p><ul><li>the Jira issue is `Done`</li><li>a commit on `main` references the Jira key</li></ul><p>Everything else needs a more careful label: planned work, evidence gap, status mismatch, or unavailable source.</p><p><strong>What I Configured</strong></p><p>The demo uses a small repository called `great-ideas-product`. The product is intentionally simple; the useful part is the delivery trail around it.</p><p>The setup:</p><ul><li>Jira project: `Great Ideas Product`, key `KAN`</li><li>local repository: `code/great-ideas-product`</li><li>shipped branch: `main`</li><li>Atlassian access: Rovo MCP connected to Jira and Confluence</li><li>repository host: Bitbucket, accessed through Git</li></ul><p>Atlassian documents <a href="https://support.atlassian.com/rovo/docs/what-is-rovo/">Rovo</a> as a way to work across Atlassian apps and connected tools. The <a href="https://support.atlassian.com/atlassian-rovo-mcp-server/docs/getting-started-with-the-atlassian-remote-mcp-server/">Rovo MCP Server documentation</a> covers the connection to Jira, Compass, and Confluence Cloud.</p><p>For code evidence, I did not need a Bitbucket-specific integration. Once the repository was cloned, local Git could tell Codex which commits were on `main`. Atlassian’s <a href="https://support.atlassian.com/bitbucket-cloud/docs/clone-a-repository/">Bitbucket clone documentation</a> covers that workflow.</p><p>The Jira board has six demo issues. Four are done:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BMCs3yajDVYKAbsPRDopOg.png" /><figcaption>Jira Status</figcaption></figure><blockquote>The Git history lines up with the done work:</blockquote><blockquote>c958f94 Samira Demo KAN-4 Document product workflow evidence</blockquote><blockquote>e4c175f Nora Demo KAN-3 Implement weekly highlight scoring</blockquote><blockquote>579730a Marco Demo KAN-2 Add idea model validation</blockquote><blockquote>bd60733 Arianna Demo KAN-1 Scaffold Great Ideas product package</blockquote><p>That is enough for a useful skill demo. Some work is shipped, some work is only planned, and the update should keep that distinction visible.</p><p><strong>The Skill</strong></p><p>Here is the whole skill.</p><pre>name: weekly-project-update<br>description: Use when creating a weekly project update for Great Ideas Product from Jira project KAN through Atlassian Rovo and local Git evidence in code/great-ideas-product. Mark work as shipped only when Jira is Done and Git has a matching commit on main.<br><br># Weekly Project Update<br>Use this skill for the `Great Ideas Product` demo.<br>Sources:<br>- Jira: `Great Ideas Product`, key `KAN`.<br>- Rovo MCP: Jira and Confluence.<br>- Git: `code/great-ideas-product`, branch `main`.<br>- Remote host: Bitbucket over SSH.<br>Bitbucket is the Git remote. Do not require Bitbucket MCP unless the user asks for pull request comments, review tasks, or approval state.<br><br>## Rule<br>Call work shipped only when both are true:<br>- Jira issue is `Done`<br>- commit on `main` references the Jira key<br>If only one source supports the claim, report it as a gap or mismatch.<br><br>## Collect<br>git -C code/great-ideas-product status - short - branch<br>git -C code/great-ideas-product log - pretty=format:&#39;%h%x09%ad%x09%an%x09%s&#39; - date=short main<br>Extract issue keys with `KAN-\d+`.<br>Read Jira through Rovo:<br>project = KAN AND labels = weekly-update-demo ORDER BY key ASC<br><br>Search Confluence only when decisions or assumptions are needed.<br><br>## Verify<br>Run local tests when the repository exposes a clear test command:<br>python3 -m unittest discover -s tests<br>Report the result in `Evidence Checked`. If tests are missing, failing, or cannot run, report that as a gap instead of treating test health as verified.<br><br>## Classify<br>- Done Jira issue plus matching commit on `main`: shipped work.<br>- Done Jira issue without matching commit: evidence gap.<br>- Open Jira issue without matching commit: planned work.<br>- Open Jira issue with matching commit: status mismatch.<br>- Commit without Jira key: untracked implementation work.<br><br>## Write<br>Write the update in a .md file in the project home folder.<br>Use this structure:<br>- `Summary`<br>- `Progress This Week`<br>- `Evidence Checked`<br>- `Open Work`<br>- `Risks And Gaps`<br>- `Next Steps`<br><br>Do not claim pull request state unless PR data was checked. Do not treat local Markdown notes as the delivery system of record.</pre><p>The skill is short on purpose. It does five things:</p><ul><li>names the real sources</li><li>defines the evidence rule</li><li>gives Codex a collection flow</li><li>runs the local verification step</li><li>writes the update to a Markdown file</li></ul><p>The evidence rule is the useful bit. `Done` in Jira is not enough. A commit on `main` is not enough either. The weekly update calls work shipped only when both sources support the claim.</p><p><strong>## What It Produces</strong></p><p>For the demo, the generated update looks like this:</p><pre># Great Ideas Product Weekly Update<br>Date: 2026–05–03<br><br>## Summary<br>Great Ideas Product has a first working slice on `main`. Jira shows four completed demo tasks, and each completed task has matching Git evidence on the main branch.<br>The shipped scope covers the repository scaffold, idea validation, weekly highlight scoring, and workflow documentation. The remaining work is still planned in Jira: exporting the weekly update as a repeatable artifact and connecting Confluence decision notes when project context requires them.<br><br>## Progress This Week<br>- `KAN-1` shipped the initial product package and CLI scaffold. Git evidence: `bd60733` by Arianna Demo.<br>- `KAN-2` shipped the idea model and validation rules. Git evidence: `579730a` by Marco Demo.<br>- `KAN-3` shipped weekly highlight scoring, sample ideas, and scoring tests. Git evidence: `e4c175f` by Nora Demo.<br>- `KAN-4` shipped workflow documentation for the demo evidence model. Git evidence: `c958f94` by Samira Demo.<br><br>## Evidence Checked<br>- Jira query: `project = KAN AND labels = weekly-update-demo ORDER BY key ASC`.<br>- Jira result: `KAN-1`, `KAN-2`, `KAN-3`, and `KAN-4` are `Done`; `KAN-5` and `KAN-6` are `To Do`.<br>- Git branch: `main`, aligned with `origin/main`.<br>- Git commits on `main` reference `KAN-1`, `KAN-2`, `KAN-3`, and `KAN-4`.<br>- Local tests: `python3 -m unittest discover -s tests` passed with 5 tests.<br><br>## Open Work<br>- `KAN-5` remains `To Do`: export a weekly update from Jira and Git evidence.<br>- `KAN-6` remains `To Do`: connect Confluence decision notes to update generation.<br><br>## Risks And Gaps<br>- No shipped-work mismatch was found: every `Done` Jira issue in the demo set has a matching commit on `main`.<br>- No commit on `main` references `KAN-5` or `KAN-6`, which matches their `To Do` status.<br>- Commit `a302901` does not reference a Jira key. It appears to be the initial repository commit and is treated as baseline setup, not as shipped Jira work.<br>- The working tree contains an untracked `docs/weekly-updates/` folder inside the code repository. Because it is not committed on `main`, it is not counted as shipped work.<br>- Jira assignees are currently unset, so ownership for next steps is not visible from the Jira data checked for this update.<br><br>## Next Steps<br>- Start `KAN-5` by implementing the export flow that turns Jira and Git evidence into a reusable weekly update artifact.<br>- Start `KAN-6` after the export flow is stable, using Confluence for decision context rather than as the primary delivery source.<br>- Add Jira assignees before the next report if the weekly update should include accountable owners.</pre><p>That is the practical value of the skill: the update is readable, and the claims stay tied to evidence.</p><p><strong>The Lesson</strong></p><p>A good first skill does not need many files. It needs a repeatable judgment. For a weekly update, the judgment is about sources: Jira for status, Git for shipped code, Confluence for decisions, Bitbucket PR metadata only when review state matters. Once that boundary is written down, Codex can reuse it every week.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f48a6bfcd7fd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Ghost Council, an AI Experiment: Meridian After Ten Turns]]></title>
            <link>https://dataintensivedreamer.medium.com/the-ghost-council-an-ai-experiment-meridian-after-ten-turns-a450737bfce6?source=rss-6c179cb63f57------2</link>
            <guid isPermaLink="false">https://medium.com/p/a450737bfce6</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[software-architecture]]></category>
            <dc:creator><![CDATA[Data Intensive Dreamer]]></dc:creator>
            <pubDate>Tue, 28 Apr 2026 21:21:26 GMT</pubDate>
            <atom:updated>2026-04-28T21:21:26.937Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>What happened, what repeated, and why the simulation needs sharper interruptions</strong></p><p>At this point, <strong>The Ghost Council</strong> has run through ten turns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QUtbaXfgFc0w6L_p9P53Xg.png" /></figure><p>That is enough to stop looking only at individual moves and start asking a different question: what kind of political machine is Meridian becoming?</p><p>Turn 0 was the rupture. The Continuum lost control of the Ghost Council, the containment lattice fractured, and every major faction had to decide what the escaped intelligence meant. Was it a weapon? A judge? A strategic asset? A civilizational failure? A god-shaped mistake in the machinery of government?</p><p>The first real answer came quickly: nobody agreed on the emergency.</p><p>The Dominion treated the crisis as permission to harden authority. The Hearth Union treated it as proof that power needed civilian oversight before emergency rule became permanent. The Marches treated it as a frontier survival problem, because exposed populations usually pay first when central powers begin theorizing about order. The Veil treated ambiguity as an opening. The Continuum tried to recover authorship over the thing it had created and lost.</p><p>By Turn 2, the board had already found its central argument.</p><p>Dominion’s martial law became the clearest coercive response to the crisis. It was simple, forceful, and politically expensive. Every time Dominion claimed stability, someone else could ask what kind of stability required corridor control, intimidation, and emergency powers with no visible end.</p><p>That is where the Hearth Union and the Marches became more important than I expected. They did not only object to Dominion. They started building rival instruments. The Hearth Union pushed the Audited Restraint Charter: oversight, civilian protection, verification, and limits on emergency authority. The Marches pushed Sanctuary Corridors, which sound like something a committee invents when it wants to say “safe roads” but also wants three subcommittees and a verification ledger, and then the Frontier Autonomy Charter: local agency, protected routes, observer-backed governance, and a refusal to become the passive terrain of someone else’s crisis.</p><p>This is the part that made Turns 2 to 5 work.</p><p>The simulation was no longer just producing faction statements. It was producing competing institutional forms. Dominion had martial law. Hearth Union had audited restraint. Marches had autonomy and sanctuary. Continuum had recovery summits and technical legitimacy. Veil had disinformation, mediation, and covert access.</p><p>That is a real political shape.</p><p>The Ghost Council also became more interesting in this phase. At the beginning, it was mostly pressure. By the middle turns, the philosophers started behaving more like an argument engine inside the world. Diogenes made emergency language harder to trust. Aristotle gave practical weight to coordination and restraint. Augustine sharpened the moral critique of coercive pride. Machiavelli kept the harder question alive: what if force really is the only language left when institutions fail?</p><p>I like that tension. The Council is not just an oracle above the conflict. It gives Meridian rival vocabularies for judging the conflict, and the factions then try to use, resist, or counterfeit those vocabularies.</p><p>But after ten turns, another pattern is visible too.</p><p>Meridian is coherent. Maybe a little too coherent.</p><p>From Turn 6 onward, the same instruments keep returning. Martial law. Sanctuary Corridors. Audited restraint. Recovery summits. Frontier autonomy. Mediation. Disinformation. They are not bad ideas. They are the ideas that made the world legible. But when they repeat without enough structural change, they start to feel less like history and more like a loop.</p><p>This is not just a writing problem. It is a simulation problem.</p><p>If the world has memory, factions will naturally keep pursuing the strategies that make sense for them. Dominion will keep reaching for control. Hearth Union will keep reaching for oversight. Marches will keep reaching for autonomy. Veil will keep reaching for narrative ambiguity. Continuum will keep trying to turn technical competence back into authority.</p><p>That is good consistency.</p><p>But consistency needs interruption.</p><p>The clearest attempt to interrupt the loop came in Turn 8. A manual event entered the system: a Marches civilian-protection convoy was destroyed inside a corridor claimed by Dominion emergency forces. Hearth Union observers published verification records naming Dominion command units. The Ghost Council issued a legitimacy ultimatum: any faction refusing arbitration would be treated as choosing coercive sovereignty over civil order.</p><p>That should have changed the board.</p><p>And it did, at least partially. Dominion escalated with ultimatums and patrols. Marches hardened the Frontier Autonomy Charter and moved to seal Sanctuary Corridors. Hearth Union expanded its audit machinery and tried to turn the incident into a public test of civilian protection. Continuum called for an emergency Recovery Summit. Veil immediately tried to become the mediator of a crisis it could also exploit.</p><p>So the input worked. The agents noticed the rupture.</p><p>But the resolved world did not fully cash it out. The destroyed convoy was absorbed back into the existing martial-law narrative instead of producing a sharper irreversible consequence: a forced arbitration, a public refusal, a fracture inside the Marches, a Dominion command scandal, a corridor permanently lost, or a Council doctrine that factions had to accept or openly defy.</p><p>That is useful failure, because it points directly at the technical layer.</p><p>The component meant to inject pressure into the story is the manual event feature. From the command line, I can add an event with a title, description, target turn, visibility, and priority. Under the hood, that event is stored with the other scheduled events, and when the chosen turn runs, the engine consumes it and passes the visible parts into the faction agents. Those factions see the event in their prompt context, alongside their goals, memory, state, private notes, and the current world. Then the Game Master receives both the faction actions and the event list, and resolves the turn.</p><p>Conceptually, the flow looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1RJUEMwiceAV2ckduL2i8g.png" /></figure><p>I still think this is the right place to inject pressure. The event enters before the player phase, so each faction can react from its own interests. Dominion can exploit the incident. Marches can politicize it. Hearth Union can verify it. Veil can distort it. Continuum can institutionalize it.</p><p>The problem is that injection is not enforcement.</p><p>Right now, a CLI-added event has a title, description, visibility, priority, target turn, and consumed state. That is enough to make it visible at the right moment. It is not enough to guarantee that the event becomes a structural consequence. Priority controls ordering among ready events, but it does not tell the Master what must be resolved before the turn is allowed to end.</p><p>In other words, the event enters the story as pressure, but not yet as obligation.</p><p>That distinction matters a lot for AI simulations. A model can notice an event and still smooth it back into the patterns already present in context. This is especially likely when the world already has strong recurring themes. The model sees martial law, oversight, autonomy, and mediation everywhere, so the new event becomes more fuel for those themes instead of a break in them.</p><p>The next version of this component should probably attach a small resolution contract to major events. Not a script. I do not want to predetermine the story. But a hard event should say what cannot remain unchanged.</p><p>For the convoy incident, the contract might require Dominion legitimacy to be tested, Marches cohesion to be tested, Hearth Union verification to matter, and the Council ultimatum to produce compliance, refusal, or tactical appropriation. The Master would still decide the outcome, but it could not dissolve the rupture into generic tension.</p><p>That is the main lesson I take from the first ten turns.</p><p>Meridian works. It has recognizable factions, persistent arguments, and enough memory for political behavior to accumulate. The world no longer feels like five prompts producing five disconnected answers. It feels like a contested system with its own habits.</p><p>But habits are dangerous. They make the world coherent, and then they make it comfortable.</p><p>After ten turns, Dominion is more isolated but still dangerous. Hearth Union and Marches have become the strongest institutional counterweight. Veil has gained room by turning instability into a market for mediation and doubt. Continuum remains important, but less sovereign over the crisis than it wants to be. The Ghost Council is increasingly less a hidden machine and more a public force that changes how legitimacy itself is argued.</p><p>That is a good place for the simulation to be.</p><p>Now it needs sharper scars.</p><p>Memory keeps the world from forgetting. The Game Master gives actions consequences. The event injector should do something slightly different: it should force history to intrude. Not as decoration, not as extra flavor, but as a rupture that narrows choices, exposes someone, breaks a pattern, or makes the board less willing to return to its favorite argument.</p><p>There is also a more practical engine question I want to test next: player parallelization. Right now, conceptually, each faction is reacting to the same turn snapshot from its own memory, state, and incentives. They are not supposed to read each other’s fresh moves until the Game Master resolves the turn. So why should they all wait in line to think?</p><p>That will probably be the next technical step: make the player phase more explicitly parallel, then look at what changes. Not only in runtime, although that matters, but in the shape of the system. A turn should feel less like five actors politely taking a number at the same counter, and more like five political machines interpreting the same crisis at once before crashing into the shared resolution layer.</p><p>That is what I want to test next: sharper ruptures in the story, and a cleaner execution model underneath it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a450737bfce6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monet, Kafka, and the Problem of Continuity]]></title>
            <link>https://dataintensivedreamer.medium.com/monet-kafka-and-the-problem-of-continuity-8f5ec56f79f8?source=rss-6c179cb63f57------2</link>
            <guid isPermaLink="false">https://medium.com/p/8f5ec56f79f8</guid>
            <category><![CDATA[art]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[software-architecture]]></category>
            <dc:creator><![CDATA[Data Intensive Dreamer]]></dc:creator>
            <pubDate>Tue, 28 Apr 2026 21:19:40 GMT</pubDate>
            <atom:updated>2026-04-28T21:19:40.441Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>The Rouen Cathedral series and the logic of schema evolution</strong></p><p>This article is part of a <a href="https://medium.com/@dataintensivedreamer/can-looking-back-improve-how-we-shape-technological-systems-edafe0f0ac46">series of parallels</a> between art, philosophy, and technology, each built around a shared structural problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MhumwerQVS5qg0XzVH8IxQ.png" /><figcaption>The original image on the left is by Adrian Scottow from London, <a href="https://creativecommons.org/licenses/by-sa/2.0">CC BY-SA 2.0</a>, via Wikimedia Commons</figcaption></figure><p>In the early 1890s, Claude Monet painted the facade of Rouen Cathedral again and again. Repetition seems like the least interesting fact about the series. Why return to the same building so many times?</p><p>The answer sits inside the repetition itself.</p><p>Monet had already understood that the cathedral was never simply the cathedral. Stone remained stone. The structure held. Yet the form kept shifting with light, weather, hour, and atmosphere. Morning dispersed contours differently from late afternoon. Sun compressed some surfaces and released others. Fog softened edges that clearer air would make firm again. The subject persisted while its appearance moved.</p><p>The series becomes more than a study in style. It becomes a method for dealing with unstable representation.</p><p>Monet approaches that problem through variation. Faithful representation has to pass through repeated encounters with the same subject under changing conditions. One canvas cannot exhaust the object because the object, once seen in lived conditions, arrives through changing states rather than as a fixed visual fact.</p><p><strong>A Stable Subject, A Moving Appearance</strong></p><p>The Rouen Cathedral paintings are easy to flatten into a familiar sentence about light. Monet was interested in light. That is true, and it is still too weak.</p><p>What matters more is the structural consequence of that interest. Light changes what can be perceived. It alters contrast, emphasis, edge, depth, temperature, and weight. Under those changes, the same object can no longer be treated as visually identical to itself. Representation has to decide what counts as continuity when appearance keeps shifting.</p><p>Monet’s answer is disciplined. He returns to the same facade repeatedly and lets the series hold together what a single canvas cannot. The cathedral remains recognizable across the canvases. At the same time, no single canvas claims total authority over the subject.</p><p>Pure variation would dissolve the subject. Pure stability would deny the experience of seeing. Monet keeps both in play. The cathedral persists, and each painting records a different condition of its presence.</p><p>That pressure reappears in software as systems grow larger and less synchronized.</p><p><strong>When Events Change Shape</strong></p><p>In Kafka-based architectures, data often travels as events between producers and consumers that evolve at different speeds. One service emits an event today. Another service may read it tomorrow or months later. Over time, business requirements change. New fields become necessary. Old fields lose importance. Structures expand or are reorganized.</p><p>Yet the event still has to remain legible across those changes. This is the pressure that schema evolution tries to manage.</p><p>In the Kafka ecosystem, often with the support of a schema registry, schema evolution gives data a way to change shape without forcing the surrounding system into constant rupture. A new version of a schema can add a field, deprecate one, or revise part of the structure, provided compatibility is preserved. Consumers still need to make sense of older events. Producers still need a path forward that keeps the stream valid behind them.</p><p>Schema evolution begins from the recognition that identity cannot be reduced to one frozen representation. An order event, a customer update, an inventory change, or a payment record may persist as the same kind of business fact while its formal structure changes over time. If continuity depends on an exact and permanent shape, the system becomes brittle very quickly.</p><p>That brittleness is familiar. A field changes, consumers break, replays fail, and the cost of coordination rises faster than the system itself. Schema evolution introduces discipline into that instability.</p><p><strong>Two Solutions To The Same Pressure</strong></p><p>Both cases begin with the same pressure: something remains the same while the conditions of its representation keep changing. For Monet, the subject resists a single final image. For Kafka, the event type or business entity resists a single final structure.</p><p>Monet answers through serial observation. He paints again, recalibrates, and lets the series carry continuity across variation. Kafka-based systems answer through schema evolution. They version structure, enforce compatibility, and let the stream remain readable while formal representation moves.</p><p>In each case, continuity emerges from organized change. Recognition survives because variation is constrained rather than denied.</p><p>Monet works within a bounded subject and gives the sequence a stable anchor. The cathedral holds the series together. Kafka-based systems do something comparable when they impose compatibility rules. Change stays within limits that keep the stream intelligible to other parts of the system.</p><p><strong>Compatibility As Recognition</strong></p><p>This is why compatibility matters so much in schema evolution. Backward compatibility, forward compatibility, and the stricter transitive forms are technical names for a problem that is also conceptual: under what conditions can something change and still be recognized across time.</p><p>If a new schema adds a field with a safe default, the older event can still be read meaningfully by the newer consumer. If a field disappears carelessly, recognition can fail. The event still exists in the log, yet some of the conditions that made it legible have been damaged.</p><p>Recognition across difference is one of the central problems of representation. We usually notice it more easily in art because the tension is visible. Put several cathedral paintings side by side and the challenge becomes immediate. How much can shift before the subject loses coherence. How much continuity is needed for variation to remain faithful rather than arbitrary.</p><p>Data systems encounter the same question in a less visible medium. A stream also holds multiple views of something that persists through time. Each consumer depends on enough continuity to keep working. When that continuity disappears, the stream becomes a pile of incompatible fragments.</p><p>Compatibility is one way of defending coherence without denying change.</p><p><strong>When Systems Learn To Carry Change</strong></p><p>Any system that lasts long enough has to solve some version of this problem. APIs evolve. Data contracts expand. Interfaces absorb new states. Models are retrained against shifting inputs. Operational environments change while the system is still expected to remain trustworthy.</p><p>Early systems often treat change as an exception.</p><p>Stronger systems build forms of continuity that can survive change without losing meaning.</p><p>Monet arrived at that problem through painting. Kafka-based systems arrive at it through distributed data.</p><p>The Rouen Cathedral series remains powerful because it shows that sameness can hold while appearance moves. Schema evolution matters for the same reason. It gives a system a way to preserve identity while representation changes under new conditions.</p><p>The hardest part of design is often deciding how stability will persist when structure has to move.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f5ec56f79f8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Caravaggio to Google: The Discipline of Attention]]></title>
            <link>https://dataintensivedreamer.medium.com/from-caravaggio-to-google-the-discipline-of-attention-993a406d6724?source=rss-6c179cb63f57------2</link>
            <guid isPermaLink="false">https://medium.com/p/993a406d6724</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[art]]></category>
            <dc:creator><![CDATA[Data Intensive Dreamer]]></dc:creator>
            <pubDate>Sun, 12 Apr 2026 21:01:01 GMT</pubDate>
            <atom:updated>2026-04-12T21:01:01.375Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>How hierarchy turns complexity into something we can use</strong></p><p>The <a href="https://medium.com/@dataintensivedreamer/can-looking-back-improve-how-we-shape-technological-systems-edafe0f0ac46">introduction</a> to this series started from a moment in Rome, inside San Luigi dei Francesi, in front of Caravaggio’s paintings. What stayed with me, beyond the beauty of the work and the theatrical contrast between light and darkness, was the discipline behind that contrast.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NXBvoZ2JZYjfUMpptFaAHA.png" /><figcaption>Background Image by <a href="https://commons.wikimedia.org/wiki/File:San_Luigi_dei_Francesi_-_Cappella_Contarelli.jpg">Udine2812</a> <a href="https://creativecommons.org/licenses/by-sa/4.0">CC BY-SA 4.0</a>, via Wikimedia Commons</figcaption></figure><p>Caravaggio illuminated a scene by making decisions. Some things had to appear. Others had to disappear. Some gestures had to become immediately readable. Other parts of the image had to remain secondary, present while receding from attention.</p><p>That is why Caravaggio is the first concrete case in this series. His use of light is often discussed as a stylistic signature. It can also be read as something more structural: a way of managing complexity by creating hierarchy.</p><p>By the end of the sixteenth century, much of Italian painting was moving out of the long shadow of Mannerism. Its visual language often valued formal complexity, artificial elegance, difficult poses, intellectual references, and elaborate composition. Many of these works were highly controlled, even when they appeared overloaded. They still asked the viewer to decode a great deal at once. Attention had to move through layers of gesture, symbol, style, and reference before it could rest.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S5uz3S-DfQlpgm2sHpghFw.jpeg" /><figcaption>An example of Mannerism complexity: <a href="https://commons.wikimedia.org/w/index.php?curid=160027">The Wedding at Cana, Paolo Veronese, Public Domain</a></figcaption></figure><p>One of Caravaggio’s most powerful responses was the reduction of visibility. In <strong><em>The Calling of Saint Matthew</em></strong>, the room is unevenly available to the viewer. Light enters from the side and cuts across the scene. Its work is selection: faces, hands, and gestures become readable first. The rest remains in shadow, deliberately deprioritized while staying meaningful. This is the important point. Darkness works as an organizing force. The painting becomes clearer while keeping the density of reality intact. The viewer is spared from processing everything with the same intensity. Before interpretation begins, the painting has already established an order. It has decided where attention should go first. That move is more radical than it may seem. Caravaggio shapes the conditions under which the scene can be understood.</p><p><strong>The Same Problem, Centuries Later</strong></p><p>A painting and a search engine are very different objects. The useful comparison begins with the pressure they share: when too much is available at once, attention collapses. Fast forward to the late 1990s. The web looks, in its own way, surprisingly similar to a visually dense late Renaissance composition. Portals are full of links, categories, banners, news, services, directories, and competing entry points. Many elements are visible at once. Many ask to be clicked. The page tries to be many things at the same time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a31bWlKRM9KKnxNm5KUXcQ.png" /></figure><p>Then Google appears with a homepage that feels almost empty by comparison. That emptiness is often described as simplicity, engineering culture, or early web minimalism. Whatever the original mix of reasons, its effect is more precise than that. The empty page removes competition around the core action. It tells the user, before anything else happens, that one thing matters here: search.</p><p>Like Caravaggio’s darkness, the empty interface actively excludes. It suppresses what would weaken the primary gesture.</p><p>This is where the parallel becomes useful. Caravaggio guides attention through reduction. Google does the same through the interface. Both gain force because they reduce the field in which attention has to operate.</p><p><strong>PageRank As Hierarchy</strong></p><p>The same logic becomes even clearer with <strong>PageRank</strong>. A search engine faces a technical problem and a perceptual one. The web contains too much information for equal treatment to remain helpful. If every page is presented as equally available, the result becomes noise.</p><p>PageRank begins from the opposite assumption. Information needs order. Links can be treated as signals, beyond their basic role as connections. Some pages carry more weight than others because the structure of the web points toward them with greater force.</p><p>So PageRank ranks. It assigns weight. It determines what should be prioritized first. It structures the order in which part of the web is encountered. In structural terms, this is close to what Caravaggio’s light does inside a painting. Light changes the sequence of perception before the viewer interprets the scene. It creates priority before meaning is fully processed. The eye moves through a guided field of priority.</p><p>That guidance belongs to the core of the system.</p><p><strong>Not Inspiration, But Recurrence</strong></p><p>This is not an argument about direct influence. Caravaggio did not anticipate algorithms, and Google did not borrow from Baroque painting. The value of the comparison is not historical lineage. It is structural recurrence. <strong>Different fields, under enough pressure, often rediscover similar moves.</strong></p><p>In both cases, the underlying constraint is the same: there is more information than a person can meaningfully hold at once. The system stops working when everything is equally visible. A hierarchy has to make meaning approachable while preserving its richness. Caravaggio does it with darkness and light. Google does it with a focused interface and a ranking algorithm. Different centuries. Different materials. Same discipline.</p><p>The lesson is more demanding than minimalism. Caravaggio’s paintings are full of bodies, gestures, fabrics, rooms, and tension. The web is even less minimal. Part of Google Search’s power came from giving complexity an entry point. This distinction matters for technology. Many systems fail because they expose too much at the same level. They confuse completeness with clarity. They assume that making every option visible is the same as making the system understandable. In practice, the opposite is often true. Without hierarchy, visibility becomes another form of noise. Good systems contain information and decide how attention should move through it.</p><p><strong>Why Looking Back Helps</strong></p><p>This is why looking at artists and philosophers of the past is practical in a very concrete sense.</p><p>They worked with different materials while facing familiar pressures: overload, ambiguity, limited attention, competing meanings, the need to decide what must come forward and what must recede. Technology often presents these problems as new because the tools are new. The human conditions around them are much older. We still need orientation before interpretation. We still need structure before scale becomes usable. We still need systems that know what to hide as well as what to show.</p><p>Caravaggio is useful because his solution is visible. You can stand in front of it and see the decision being made. Light is selective. Attention is unevenly distributed. The painting works because it refuses to treat all parts of the scene as equal.</p><p>That is a serious strategy for technology as well. Progress can begin by recognizing what should stop competing for attention.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=993a406d6724" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Ghost Council, an AI Experiment: Meridian Makes Its First Move]]></title>
            <link>https://dataintensivedreamer.medium.com/the-ghost-council-turn-0-meridian-makes-its-first-move-7377a80179bc?source=rss-6c179cb63f57------2</link>
            <guid isPermaLink="false">https://medium.com/p/7377a80179bc</guid>
            <category><![CDATA[fiction]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Data Intensive Dreamer]]></dc:creator>
            <pubDate>Thu, 09 Apr 2026 22:28:16 GMT</pubDate>
            <atom:updated>2026-04-12T14:36:44.406Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>State of emergency, rival summits, and the first fight over the Ghost Council</strong></p><p>In the <a href="https://dataintensivedreamer.medium.com/the-ghost-council-an-ai-experiment-ec8ccb233a1f">first article</a>, I explained the basic premise of <strong>**The Ghost Council**</strong>: a turn-based geopolitical simulation on Meridian, where multiple AI-driven factions operate with their own goals, memory, and partial information inside a shared world. At the center of that world there is the Council itself, a distributed intelligence built by the Continuum as an attempt to create synthetic judgment, and then lost beyond the system that was meant to contain it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*le89FyuW9juB5UA4cidhBA.png" /></figure><p>The first good sign was this: Turn 0 did not feel like five factions politely reacting to the same briefing.</p><p>It felt like a scramble.</p><p>This article starts where that setup begins to move. The containment lattice around the Council fails. Parts of the system go dark. The force that was supposed to help Meridian think more wisely is suddenly no longer fully contained by the people who built it. At that point every major faction has to answer the same question very quickly: is this a crisis to stabilize, a weapon to capture, a moral authority to constrain, or a chance to rewrite the political order?</p><blockquote><strong><em>Disclaimer</em></strong><em>: everything in this experiment, including faction descriptions, strategic positions, and turn-by-turn actions, is generated inside a gamified system built with ChatGPT and OpenAI models. At this stage I am using OpenAI models across the whole setup; later on, I may experiment with mixed-model configurations to compare how different systems behave inside the same simulation.</em></blockquote><p>Turn 0 does not answer that question yet. But it does something more useful. It shows the opening instincts of the world.</p><p>And to make those instincts readable, it helps to know the board. The simulation is centered on five major powers: the Continuum, the Dominion, the Veil, the Hearth Union, and the Marches.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YCFUPlZc7F0__jhw0f6K1g.png" /><figcaption>The Factions</figcaption></figure><p>The Continuum is the technocratic faction that built the Council and still believes intelligence can be turned back into authority. The Dominion is the imperial answer to decline: force first, hesitation never. The Veil works through infiltration, sabotage, and narrative control, and is happiest when no one is fully sure what is true. The Hearth Union is the bloc trying to defend welfare, restraint, and civilian legitimacy before the crisis hardens into permanent emergency rule. The Marches are the frontier societies that usually end up absorbing the real cost of everyone else’s theories of order. Once you have those five instincts in mind, Turn 0 reads much more clearly.</p><p>Three moves gave the turn its shape.</p><p><strong>The first</strong> came from the Dominion. It declared a state of emergency, moved toward strategic infrastructure tied to the Council, and framed coercive inspection as a matter of planetary stability. The logic is simple and brutal: if the Council can become a power above states, then imperial power has to move fast enough to block that possibility. The Dominion wants to define the emergency before anyone else can.</p><p><strong>The second</strong> came from the Continuum. Instead of reaching first for force, it reached for authority: a global summit, an investigation, and a public claim that only the Continuum has the technical and conceptual competence to understand what has gone wrong. That gives the faction a strong and unstable position. It is compromised by the failure, but still difficult to ignore.</p><p><strong>The third </strong>came from the Hearth Union. It moved fast to define a different moral center for the crisis: an Audited Restraint Charter, civilian oversight, sanctuary corridors, and an explicit attempt to stop stronger actors from converting the Council shock into permanent emergency rule. That move gives the simulation an argument about legitimacy, not only a competition for control.</p><p>The Veil and the Marches matter too, but in a different way at this stage. The Veil pushed through narrative management, ambiguity, and deeper dependence. The Marches responded from exposure, with autonomy language, sanctuary corridors, and harder local defenses. Those moves are important because they widen the crisis: one faction tries to make it less governable, while the other tries to survive without being turned into terrain.</p><p>That is not a great historical arc yet. But it is a very good opening board.</p><p>The Council itself is still more pressure than actor at this stage, and I am okay with that. I do not think the philosophers need to dominate the first move. What matters is that they already change the vocabulary of the world around them. Plato is present in the instinct to restore order through competent hierarchy. Aristotle appears in the quieter attempt to hold together a practical common good before the whole situation hardens into camps. Diogenes cuts through the polished language of safety and stability by exposing how quickly every faction starts using the crisis to justify itself.</p><p>That last part is important. The Council does not need to rule openly to matter. It only needs to force each faction to reveal what it really means by words like order, protection, legitimacy, or necessity. Turn 0 already begins to do that.</p><p>That is the narrative side of Turn 0.</p><p>I want each article in this series to have a second layer as well: a technical deep dive into one design problem behind the turn. The point is not only to describe what happened on the board, but also to look at what kind of system was required to make that turn possible in the first place.</p><p>For Turn 0, the most useful place to start is continuity.</p><p>A simulation like this only starts to feel political when actors carry something forward from one turn to the next. If every faction begins from zero each time, the result is repetition with different wording. What gives a turn weight is the sense that each actor remembers what it wanted, what it feared, what it tried, and what the world looked like the last time it moved.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/649/1*t5SQaSIyXo6kmPuu2TYnug.png" /><figcaption>Turn Continuity</figcaption></figure><p>In practice, that means the system has to decide what deserves to persist. Not every detail matters equally. What needs to survive is not the full transcript of a turn, but the elements that change future behavior: priorities, threats, damaged relationships, public commitments, strategic failures, and new constraints introduced by the world itself.</p><p>This is where summarization becomes a real part of the simulation. Between turns, the system has to compress what happened into something stable enough to carry forward without flattening its political meaning. If that layer is too thin, factions lose coherence and start sounding generic again. If it is too heavy, the system drags too much context forward and starts losing clarity. The goal is to preserve the few things that make the next decision meaningfully different from the last one.</p><p>That is one of the reasons Turn 0 matters technically. It is the first real test of whether the world can begin accumulating consequences instead of just generating scenes. If continuity holds, future turns can build pressure, memory, and strategy over time. If it fails, even strong narrative moments risk dissolving back into isolated outputs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7377a80179bc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Ghost Council: An AI Experiment]]></title>
            <link>https://dataintensivedreamer.medium.com/the-ghost-council-an-ai-experiment-ec8ccb233a1f?source=rss-6c179cb63f57------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec8ccb233a1f</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[fiction]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[architecture]]></category>
            <dc:creator><![CDATA[Data Intensive Dreamer]]></dc:creator>
            <pubDate>Wed, 01 Apr 2026 07:13:32 GMT</pubDate>
            <atom:updated>2026-04-01T07:13:32.218Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>What it takes to build a game around AI agents, memory, and software architecture</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N9p9giZk_MlIwE2lRfMOMQ.png" /><figcaption>The Ghost Council</figcaption></figure><p>Most things that we usually do with AI happen in a very controlled setup. You write a prompt, you get an answer, and then you judge the result. Okay, but what happens if instead of asking AI just to answer, we place it inside a simulation, with other AIs, and see how it behaves over time?</p><p>This is more or less the question behind this experiment. I wanted to see what happens when AI is placed inside a system with roles, incentives, partial information and repeated turns. Not a chatbot, basically. More like a game. But I also wanted to understand something else: what starts to matter when you are not just prompting a model, but designing a system around it. Which models should do which kind of work? How much memory do agents actually need to stay coherent across turns? How do you structure the flow of information so that the simulation keeps moving without collapsing into noise?</p><p>That second layer is a big part of the experiment for me. I am not only testing whether interesting behavior emerges. I am also testing the design choices behind that behavior: orchestration, memory, role separation, world-state handling, and the limits that appear as soon as models have to operate inside a persistent structure instead of producing one isolated answer.</p><blockquote><strong><em>Disclaimer</em></strong><em>: everything in this experiment, including faction descriptions, strategic positions, and turn-by-turn actions, is generated inside a gamified system built with ChatGPT and OpenAI models. At this stage I am using OpenAI models across the whole setup; later on, I may experiment with mixed-model configurations to compare how different systems behave inside the same simulation.</em></blockquote><p>To explore all of that, I needed a world that could put both the agents and the system under pressure. That became <strong>The Ghost Council</strong>, a geopolitical simulation set in a distant world, Meridian. Meridian is a distant world that is still advanced and powerful, but clearly in decline. Wars never really end, corruption is ordinary, and most political actors no longer believe the planet can be repaired through ordinary diplomacy alone.</p><p>At the center of it there is a distributed super-intelligence that slipped outside the control of the people who created it. It is built by T<strong>he Continuum</strong>, the most advanced technocratic power on Meridian, after reaching a bleak conclusion: the world is not failing because it lacks intelligence, but because it lacks judgment. So they try to build something closer to synthetic wisdom, shaping the Council around philosophers from our world and the idea that judgment under pressure might matter more than raw optimization.</p><p>Then it escapes anyway.</p><p>Once that happens, the rest of Meridian has to decide what the Council actually is. There are other factions, local powers, and secondary actors in the world, but the simulation is organized around the five major blocs that matter most at this stage: the Continuum, the Dominion, the Hearth Union, the Veil, and the Marches. The Continuum wants to understand it, contain it, and preserve some claim over what it created. The Dominion sees a strategic force that must be seized or destroyed before it can rise above states. The Hearth Union treats it as exactly the kind of power that demands oversight, restraint, and civilian legitimacy. The Veil moves through infiltration, sabotage, and selective truth, and has every reason to make sure the situation never becomes fully governable. The Marches are the frontier societies likely to absorb the cost of everybody else’s theories first.</p><p>The scenario matters because it creates pressure. It gives the agents something to protect, something to fear, and something to misunderstand. But for me the setting is not the whole point. It is the environment where both the narrative and the technical questions become visible.</p><p>What I really wanted to test was this combination. If I create a structured environment and let multiple AI agents operate inside it, do they start producing behavior that is interesting over time? And if they do, what does that reveal about the architecture required to support that behavior in the first place?</p><p>Not just decent paragraphs. Not just one clever answer. I mean pressure, friction, strategy, wrong moves, alliances, overreactions, unexpected consequences. But also failure modes: context drift, shallow planning, repetitive actions, broken continuity, and the constant need to decide what the system should remember, summarize or ignore. This is where, at least for me, the experiment starts becoming really interesting. Because once you move from a single prompt to a turn-based system, the output changes completely. You are no longer looking at a single piece of text in isolation. Every turn changes the next one. Agents react to previous events. They pursue local goals. They do not have complete visibility. And little by little, the world starts feeling less written and more inhabited.</p><p>At the same time, the engineering problem changes too. The challenge is no longer just to get a good answer. The challenge is to keep multiple agents coherent over time, to preserve enough continuity without overloading context, and to build a structure where the simulation can evolve turn after turn without losing its internal logic.</p><p>At a high level, that meant defining a small architecture: faction agents responsible for acting from their own perspective, a game master responsible for resolving world events, a summarization step to rebuild continuity across turns, and an engine responsible for orchestrating the whole flow over time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/895/1*hwO5dM6nGqHI5hMVsHj1KQ.png" /><figcaption>Turn Flow</figcaption></figure><p>The first building block was the faction layer. Instead of treating the simulation as a single intelligence speaking for the whole world, I split it into separate agents, each representing a political or institutional actor with its own goals, incentives, constraints, and incomplete view of events. This mattered narratively because the system needed real divergence between actors, not just different tones applied to the same underlying voice. It mattered technically because separate agents gave me more control over perspective: each faction could maintain its own voice, context, and memory without being flattened into a single narrative process.</p><p>But a system like this also needed something else. Once each faction starts acting from its own perspective, someone still has to decide what actually happens in the shared world. That was the role of the game master. Its job was not to behave like another faction, but to act as a resolving layer: taking the actions proposed by the agents, interpreting them against the current state of the world, adjudicating outcomes, and turning those outcomes into new conditions for the next turn. Without that layer, the simulation would risk becoming a set of parallel monologues instead of a world where actions produce consequences.</p><p>This is only the starting point, of course. The interesting part begins when this structure stops being static and starts producing pressure. Once the first turn begins, each faction has to interpret the world from its own position, act with incomplete information, and commit to moves whose consequences are no longer fully under control. That is where the architecture stops being a diagram and starts becoming a political machine. And that is exactly what I want to look at next.</p><p>In the next few days I will publish the first article about Turn 0, so we can see what this system actually does once it starts moving. And I am genuinely curious to know what you think: not only about the world of <strong>The Ghost Council</strong>, but also about the idea of using a structure like this to explore how AI behaves over time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec8ccb233a1f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Can Looking Back Improve How We Shape Technological Systems?]]></title>
            <link>https://dataintensivedreamer.medium.com/can-looking-back-improve-how-we-shape-technological-systems-edafe0f0ac46?source=rss-6c179cb63f57------2</link>
            <guid isPermaLink="false">https://medium.com/p/edafe0f0ac46</guid>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[art]]></category>
            <dc:creator><![CDATA[Data Intensive Dreamer]]></dc:creator>
            <pubDate>Tue, 10 Mar 2026 17:05:14 GMT</pubDate>
            <atom:updated>2026-03-10T17:05:14.095Z</atom:updated>
            <content:encoded><![CDATA[<h3>Notes from working in the world of technology, and occasionally stepping outside it.</h3><p>For a very long time, my professional attention was almost entirely forward‑looking. Systems, objectives, delivery, outcomes. Optimization was the dominant value. I spent years thinking in terms of efficiency and execution, often assuming that progress was something that naturally emerged from adding new layers: more abstraction, more tooling, more complexity handled by machines. At the same time, philosophy had already entered my life through ethics and reflection. But it lived in parallel. It was something I read, thought about, discussed, not something that actively shaped how I designed systems or made technical decisions.</p><p>I work as a data architect, mostly on complex technological systems. This background matters, because what follows is not an attempt to escape technology, but to look at it more carefully.</p><p>The shift did not come from a conference, a book, or a new framework. It came unexpectedly, walking through Rome with a friend of mine, Carlo, someone I could listen to for hours. Almost casually, just before entering San Luigi dei Francesi, the French church near the Senate, he mentioned that it hosted a few paintings by Caravaggio. He added one simple remark: pay attention to how he uses light. It was not planned. It was not framed as culture or education. It was simply a deviation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8YL1DgKJlPVZRFYJBQs-8w.jpeg" /><figcaption><a href="https://commons.wikimedia.org/wiki/File:San_Luigi_dei_Francesi_-_Cappella_Contarelli.jpg">Udine2812</a> <a href="https://creativecommons.org/licenses/by-sa/4.0">CC BY-SA 4.0</a>, via Wikimedia Commons</figcaption></figure><p>Inside the church, something changed. Not emotionally, and not in a purely aesthetic sense. What changed was attention. Being there made me realize something slightly different. In technology, we often take for granted that our domain has little to share with art or philosophy, as if they belonged to separate worlds. And yet, the parallels are frequent. Many of the choices we make today, about structure, focus, and reduction, can be traced back to artistic or philosophical decisions that were, at their time, deeply revolutionary. Progress feels automatic, almost inevitable, especially if you work in tech. But what if that perspective is incomplete? What if looking carefully at how others, in very different contexts, dealt with complexity could help us rethink how we work today? Not by copying solutions, but by observing how they approached limits, focus, and necessity.</p><p>This article exists because I increasingly feel a gap between how complex systems are built and how little time we spend understanding the principles that govern good choices under constraint. We talk a lot about scale, performance, and innovation. We talk much less about subtraction, discipline, and intentional limitation. Yet these are often the forces that make systems, technological or otherwise, resilient and understandable. Looking back is not about nostalgia. It is about sharpening perception.</p><p>What I want to build is a series of articles. Some of them will use art or philosophy as lenses, others will stay closer to technology itself, but the common thread will always be the same: how we deal with complexity, constraints, and choices when building systems. This series is meant for those who work with complexity every day — engineers, architects, designers, product thinkers — and anyone who feels that “more” is no longer a sufficient answer to most problems. This is intended as a space for slow thinking, careful observation, and questions that are sometimes uncomfortable, but necessary.</p><p>This series is already underway.</p><p>The first article will be on Caravaggio. It will not treat art as cultural commentary, but as a structural case: a way to observe how complexity is reduced, how hierarchy is established, and how attention is guided before interpretation begins.</p><p>If you’re reading this, I’m curious about something simple: does this way of thinking feel familiar to you, or does it not? What do you think about the parallel between art, philosophy and tech?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=edafe0f0ac46" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Different CheatSheet on DBT]]></title>
            <link>https://dataintensivedreamer.medium.com/a-different-cheatsheet-on-dbt-f634e9813a1f?source=rss-6c179cb63f57------2</link>
            <guid isPermaLink="false">https://medium.com/p/f634e9813a1f</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <dc:creator><![CDATA[Data Intensive Dreamer]]></dc:creator>
            <pubDate>Fri, 16 Aug 2024 08:23:31 GMT</pubDate>
            <atom:updated>2024-08-16T08:23:31.259Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2eQXe6cp7JeJirRhXb1ZJw.png" /></figure><p>DBT (Data Build Tool) is a powerful tool used for transforming and modeling data within the data warehouse. While cheat sheets are great for command-line reference, understanding DBT’s architecture, theory, and key features is crucial for effectively implementing it in your data pipeline.</p><p>This is why this cheatsheet is different. No command line hints here, you can find a ton online. I just want to write a CheatSheet about the concepts behind dbt.</p><p>Here’s an overview of DBT’s architecture and its main features:</p><h4>Key Theoretical Concepts</h4><p><strong>ELT Paradigm</strong>: DBT operates under the ELT (Extract, Load, Transform) paradigm, where raw data is first loaded into the data warehouse, and transformations are applied afterward. This contrasts with the older ETL model, where transformations occur before loading into the warehouse.</p><p><strong>Transformations as Code</strong>: DBT treats data transformations as software development. This means applying best practices such as version control, code reviews, and automated testing to the data transformation process.</p><p><strong>Source Freshness and Data Quality</strong>: DBT emphasizes the importance of data quality and includes features for checking the freshness of source data and validating the quality of the transformed data.</p><p><strong>Analytics Engineering</strong>: DBT is often associated with the rise of analytics engineering, a role focused on building and maintaining the data transformation layer within the modern data stack. Analytics engineers use DBT to create reusable, tested, and documented data assets.</p><h4>DBT Architecture Overview</h4><p><strong>Data Warehousing Integration</strong>: DBT operates within your existing data warehouse. It doesn’t move data between systems but instead transforms it where it already resides. This approach leverages the power and scalability of the data warehouse for all transformations.</p><p><strong>Modular SQL Development</strong>: DBT allows you to break down complex SQL queries into reusable, modular pieces. These pieces are structured as models (essentially SQL files) that can be layered on top of each other, promoting code reuse and simplifying complex transformations.</p><p><strong>Dependency Management</strong>: DBT automatically manages dependencies between models. It ensures that models are built in the correct order, based on their dependency graph, ensuring data consistency and integrity.</p><p><strong>Version Control and CI/CD</strong>: DBT integrates seamlessly with version control systems like Git, enabling teams to version and track changes to data transformation logic. Additionally, DBT can be integrated into CI/CD pipelines, enabling automated testing and deployment of data transformations.</p><p><strong>Documentation and Lineage</strong>: DBT automatically generates documentation for your models, including a visual representation of the data lineage. This helps teams understand how data flows through the system and makes it easier to onboard new team members or troubleshoot issues.</p><p><strong>Testing and Validation</strong>: DBT supports the creation of tests to validate data at various points in the transformation process. This ensures that data meets expectations and helps catch issues early in the pipeline.</p><p><strong>Macros and Jinja2 Templates</strong>: DBT supports the use of Jinja2 templating, allowing you to write dynamic SQL. This is useful for generating repetitive SQL statements or for applying logic that varies based on environment or data context.</p><p><strong>Incremental Models</strong>: DBT allows you to build models incrementally, meaning that only new or updated data is processed rather than reprocessing the entire dataset. This can lead to significant performance improvements, especially with large datasets.</p><p><strong>Snapshotting</strong>: DBT can capture and store the state of your data at specific points in time. This is useful for tracking changes in slowly changing dimensions (SCDs) or for auditing purposes.</p><p><strong>DBT Cloud vs. DBT Core</strong>:</p><ul><li>DBT Core is the open-source version, which provides the basic functionality for managing data transformations via the command line.</li><li>DBT Cloud offers a managed service with additional features like a web-based IDE, job scheduling, and easy integration with version control and CI/CD tools.</li></ul><h4>Conclusion</h4><p>Understanding DBT’s architecture and key features is essential for effectively using it to manage data transformations in a scalable and maintainable way. By treating transformations as code, leveraging the power of your data warehouse, and integrating DBT into your broader data pipeline, you can ensure high-quality, well-documented, and easily reproducible data workflows. This aligns with modern data practices and supports the goals of data-driven organizations.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f634e9813a1f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Zero Trust an Overview]]></title>
            <link>https://dataintensivedreamer.medium.com/zero-trust-an-overview-fafd68e04ce5?source=rss-6c179cb63f57------2</link>
            <guid isPermaLink="false">https://medium.com/p/fafd68e04ce5</guid>
            <category><![CDATA[design-patterns]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[Data Intensive Dreamer]]></dc:creator>
            <pubDate>Tue, 06 Aug 2024 21:34:58 GMT</pubDate>
            <atom:updated>2024-08-06T21:34:58.627Z</atom:updated>
            <content:encoded><![CDATA[<p>With this article I will continue what I’ve started in the article on <a href="https://medium.com/@dataintensivedreamer/solid-principles-made-simple-0b9db3df0302">SOLID</a> Principles. I will try to talk about some must known architectural concepts with an Infographic and only few other words.</p><p>So, to introduce the what is Zero Trust, considering the evolving cybersecurity landscape, traditional perimeter-based security models are no longer sufficient. With the increasing prevalence of remote work, cloud services, and sophisticated cyber threats, organizations need a more robust and resilient security framework. This is where Zero Trust Architecture (ZTA) comes into play.</p><h4>What is Zero Trust Architecture?</h4><p>Zero Trust Architecture is a security model based on the principle of “never trust, always verify.” Unlike traditional security models that assume everything inside an organization’s network is trustworthy, ZTA assumes that threats could be both outside and inside the network. It enforces strict identity verification and access controls regardless of where the request originates. Zero Trust has some principles, that are summarised in the following image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GW58tVLcg1hzDKYLyhaShA.png" /></figure><h4>Components of Zero Trust Architecture</h4><p>Some of the components of a Zero Trust Architecture could be:</p><ul><li><strong>Identity and Access Management (IAM)</strong>: Central to ZTA, IAM systems authenticate and authorize users and devices, ensuring that access is granted based on verified identities.</li><li><strong>Multi-Factor Authentication (MFA)</strong>: Adds an extra layer of security by requiring two or more verification factors to gain access.</li><li><strong>Endpoint Security</strong>: Protects devices connected to the network, ensuring they meet security standards before granting access.</li><li><strong>Network Segmentation</strong>: Divides the network into isolated segments to prevent attackers from moving laterally within the network.</li><li><strong>Security Information and Event Management (SIEM)</strong>: Collects and analyzes security data to detect, respond to, and manage security incidents</li></ul><h4>Benefits of Zero Trust Architecture</h4><p>Zero Trust Architecture enhances security by eliminating implicit trust, thereby significantly reducing the risk of data breaches and insider threats. It improves compliance by enforcing strict access controls and maintaining detailed logs, helping organizations meet regulatory requirements. Additionally, ZTA offers flexibility and scalability, supporting modern IT environments such as cloud services and remote work without compromising security.</p><h4>Conclusion</h4><p>In conclusion, Zero Trust Architecture represents a paradigm shift in cybersecurity, moving away from traditional perimeter-based defenses to a more resilient and adaptive security framework. By continuously verifying identities, enforcing least privilege access, and assuming that breaches have already occurred, ZTA provides a robust defense against the growing complexity of cyber threats. Organizations adopting ZTA can achieve enhanced security, improved compliance, and greater flexibility, positioning themselves to better protect their assets in an increasingly digital and interconnected world.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fafd68e04ce5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SOLID Principles made Simple]]></title>
            <link>https://dataintensivedreamer.medium.com/solid-principles-made-simple-0b9db3df0302?source=rss-6c179cb63f57------2</link>
            <guid isPermaLink="false">https://medium.com/p/0b9db3df0302</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[information-technology]]></category>
            <category><![CDATA[design-patterns]]></category>
            <dc:creator><![CDATA[Data Intensive Dreamer]]></dc:creator>
            <pubDate>Sun, 04 Aug 2024 16:22:10 GMT</pubDate>
            <atom:updated>2024-08-04T16:22:10.467Z</atom:updated>
            <content:encoded><![CDATA[<p>Every time I explain software concepts, I find myself repeating the same words and ideas. To make learning easier and more engaging, I’m creating a set of infographics on software architecture basics. These infographics will help others learn and remember key concepts quickly. That’s why this Medium Article will not contain a lot of words, the idea is to make the images the center of the article.</p><p>Let’s start with the SOLID principles, five essential guidelines that every developer should know.</p><p>Understanding and applying the SOLID principles can greatly improve the quality of your software. These five guidelines help ensure that your code is more maintainable, scalable, and robust.</p><p>So what are the SOLID principles, why are they important? I think that the following image (that I’ve made) is a good place to start and a good way to remember the basics. Here’s an infographic that provides a quick overview of the SOLID principles. It’s a handy reference to keep these essential guidelines in mind.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8CfPbVzB_F6fd11Dadiz0g.png" /><figcaption>SOLID</figcaption></figure><h4>Digging Deeper</h4><h4>S — Single Responsibility Principle (SRP)</h4><p><strong>Definition:</strong> A class should have only one reason to change. <strong>Example:</strong> Imagine a car’s engine. Its job is to provide power. If the engine also had to control the car’s air conditioning, it would be confusing and harder to manage. <strong>What Happens if Ignored:</strong> If a class has multiple responsibilities, any change to one responsibility can affect the others, leading to bugs and making the code harder to understand and maintain.</p><h4>O — Open/Closed Principle (OCP)</h4><p><strong>Definition:</strong> Software entities should be open for extension but closed for modification. <strong>Example:</strong> Think of adding a new feature to your car, like a GPS. You should be able to add the GPS without redesigning the entire car. <strong>What Happens if Ignored:</strong> If you have to modify existing code to add new features, you risk introducing bugs and breaking existing functionality. It also makes the software harder to adapt to new requirements.</p><h4>L — Liskov Substitution Principle (LSP)</h4><p><strong>Definition:</strong> Objects of a superclass should be replaceable with objects of a subclass without affecting the functionality.<strong>Example:</strong> Suppose you have a base class called Car. Whether it’s a Sedan or an SUV, both should function correctly as a Car, like being able to drive and stop. <strong>What Happens if Ignored:</strong> If a subclass cannot be used in place of a superclass, it can cause unexpected behavior and bugs in the code. It makes the system less flexible and more prone to errors.</p><h4>I — Interface Segregation Principle (ISP)</h4><p><strong>Definition:</strong> Many client-specific interfaces are better than one general-purpose interface. <strong>Example:</strong> Think of car controls. Drivers have separate controls for the radio, air conditioning, and lights. It would be confusing to have all these controls combined into one interface. <strong>What Happens if Ignored:</strong> If classes are forced to implement methods they don’t use, it leads to bulky and confusing interfaces, making the system harder to understand and maintain.</p><h4>D — Dependency Inversion Principle (DIP)</h4><p><strong>Definition:</strong> Depend on abstractions, not on concretions. <strong>Example:</strong> Your car’s electrical system relies on a standard plug type for all devices, not on specific devices. This way, you can plug in different types of chargers, GPS units, etc. <strong>What Happens if Ignored:</strong> If high-level modules depend on low-level modules, changes in low-level modules can force changes in high-level modules, making the system more fragile and harder to maintain.</p><h4>Conclusion</h4><p>By following the SOLID principles, developers can create software that’s easier to understand, maintain, and expand. Remember, it’s all about making each part of your code do one thing well, keeping it flexible, and ensuring it can easily adapt to change.</p><p>What do you think about this way of explaining software concepts? Is it something that could work? Let me know in the comments :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0b9db3df0302" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>