<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vanik Karslian on Medium]]></title>
        <description><![CDATA[Stories by Vanik Karslian on Medium]]></description>
        <link>https://medium.com/@vanik_kars?source=rss-42fb2c934fcb------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*u39VJBlDUgKlpAdU4nuphg.jpeg</url>
            <title>Stories by Vanik Karslian on Medium</title>
            <link>https://medium.com/@vanik_kars?source=rss-42fb2c934fcb------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 03:03:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vanik_kars/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[From YAML to a Conversation: Generating Airflow Pipelines with an LLM]]></title>
            <link>https://medium.com/@vanik_kars/from-yaml-to-a-conversation-generating-airflow-pipelines-with-an-llm-55585acfccea?source=rss-42fb2c934fcb------2</link>
            <guid isPermaLink="false">https://medium.com/p/55585acfccea</guid>
            <category><![CDATA[apache-iceberg]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[amazon-bedrock]]></category>
            <category><![CDATA[apache-airflow]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Vanik Karslian]]></dc:creator>
            <pubDate>Sun, 11 Oct 2026 20:14:34 GMT</pubDate>
            <atom:updated>2026-10-11T20:14:34.448Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nDNLKAgiaid8qUGW" /><figcaption>Photo by <a href="https://unsplash.com/@hamburgmeinefreundin?utm_source=medium&amp;utm_medium=referral">Wolfgang Weiser</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>A chatbot that writes declarative pipelines — and a validator that decides whether they are allowed to exist</h3><p>In the <a href="https://medium.com/@vanik_kars/declarative-yaml-based-airflow-pipelines-as-data-platform-capability-e7b3b555e78d">previous article</a> I made the case for treating pipeline definitions as configuration rather than code. A platform team owns a small number of blueprints — parameterised task groups that encode how ingestion is actually done here — and a data team declares pipelines against them in YAML.</p><p>The result is a file like this:</p><pre>---<br># Banking Application - Postgres to Iceberg Pipeline<br>dag_id: banking_users_to_iceberg<br>description: &quot;Extract banking users table to Iceberg in S3&quot;<br>schedule: &quot;@daily&quot;<br>start_date: &quot;2024-01-01&quot;<br>catchup: false<br>steps:<br>  ingest_users_table:<br>    blueprint: postgres_to_iceberg_blueprint<br>    table_name: users<br>    schema_name: public<br>    iceberg_namespace: banking<br>    iceberg_table_name: users<br>    s3_bucket: iceberg-warehouse<br>    s3_prefix: warehouse<br>    mode: overwrite<br>    batch_size: 100<br>    postgres_conn_id: postgres_banking</pre><p>That is an entire production-shaped pipeline: Postgres to Apache Iceberg, scheduled, retried, with a task group behind it that extracts the schema, creates the Iceberg table, pages through the source and writes Parquet. No Python, no operators, no DAG object. The platform team owns postgres_to_iceberg.py; the data team owns the eleven lines above.</p><p>This article is about what happened next, and it is deliberately not about blueprints or Airflow. It is about the layer on top: a chatbot that produces that YAML from a sentence, and the validator that stands between the model and the dags folder.</p><p>The code is at <a href="https://github.com/vanikkars/airflow-blueprint-gen-ai">vanikkars/airflow-blueprint-gen-ai</a>.</p><h3>Why bother, when the YAML is already eleven lines</h3><p>This is the fair objection, and it is worth answering honestly before the architecture: eleven lines of YAML is not a hard problem. Nobody needs an LLM to type them.</p><p>The YAML is not where the time goes. The time goes into the questions around it:</p><ul><li>Which of the config keys does this blueprint actually accept?</li><li>What is the namespace convention in this repo — banking, raw_banking, bronze?</li><li>Is there already a DAG ingesting this table under a different name?</li><li>Does transactions even exist in the source, or is it transaction?</li><li>Which Airflow connection points at that database?</li></ul><p>And the one nobody asks, because asking it requires knowing it is a question: what will this blueprint do to a numeric(15,2) column on the way to Iceberg?</p><p>Answering those means opening the blueprint’s Pydantic model, the connections provisioning script, four existing DAG files, and a psql session. The declarative layer removed the Python. It did not remove the “archaeology”.</p><p>So the benefit of the chat interface is not typing speed. It is three specific things:</p><p><strong>The context comes to you.</strong> The generator introspects the live source database, the blueprint registry, the existing DAGs and the real connection list on every session, and puts all of it in front of the model. You stop being the one who has to go and look.</p><p><strong>The hazards are stated without being asked for.</strong> Every proposal comes back with warnings derived from the actual schema — money columns that will lose exactness, PII that lands unmasked, overwrite discarding snapshot history. These are the things a reviewer catches on a good day and misses on a Friday.</p><p><strong>The iteration cost drops to a sentence.</strong> “Actually make it hourly and append instead of overwrite” revises what was just proposed. Nothing touches disk until you say so, so an unsatisfying first attempt costs nothing.</p><p>What it is explicitly <em>not</em>: a way to let the model invent infrastructure. Everything below exists to keep it from doing that.</p><h3>What one request actually does</h3><p>Here is the whole flow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AzwGRGNDYYckE1MfdcoKhQ.png" /><figcaption><em>One request, end to end. The dashed red path is the repair loop.</em></figcaption></figure><p>The rest of this article walks those five steps. The short version of the lesson is that step 4 is where the engineering is, and step 3 is the easy part.</p><h3>Step 1: grounding — the code decides what the model may know</h3><p>Everything the model is allowed to know arrives in one message, assembled by introspect.py. It has no tools. It cannot query the database, list files, or read the registry itself. That is the design: if a fact is not in the message, the model has no legitimate way to know it, and the rules tell it to refuse rather than guess.</p><p>Four blocks, all read live.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rXQLeUhdj9fxpHTERpdnug.png" /><figcaption><em>Four sources in, one message out — and the catalog arrives annotated, not just dumped.</em></figcaption></figure><p><strong>The blueprint registry.</strong> Not a hand-written list — the same registry Airflow’s loader uses, reflected for each config field’s type, default and description:</p><pre>registry = BlueprintRegistry(template_dirs=[BLUEPRINTS_DIR, DAGS_DIR])<br>registry.discover()</pre><p>Drop a blueprint into airflow/blueprints/ and the generator can use it on the next run. No list to update means no list to forget.</p><p><strong>The source catalog.</strong> Live information_schema, plus the planner&#39;s row estimate rather than a count(*) scan. The useful part is not the query — it is that each column is annotated with what the blueprint will <em>actually do to it</em>:</p><pre>if data_type not in MAPPED_PG_TYPES:<br>    notes.append(&quot;UNMAPPED -&gt; falls back to Iceberg StringType&quot;)<br>elif data_type in LOSSY_PG_TYPES:<br>    notes.append(&quot;maps to Iceberg DoubleType - loses exact precision&quot;)<br>if any(pattern in name.lower() for pattern in SENSITIVE_PATTERNS):<br>    notes.append(&quot;SENSITIVE - no masking in the blueprint&quot;)</pre><p>Which produces:</p><pre>### public.users<br>Approximate rows: ~100<br>Primary key: user_id<br>Columns:<br>  - email: character varying(255) [nullable; SENSITIVE - no masking in the blueprint]<br>  - balance: numeric(15,2) [nullable; maps to Iceberg DoubleType - loses exact precision]</pre><p>So when a proposal warns that balance loses the floating points and precision, that is not the model&#39;s instinct about money being delicate. It is a fact computed from this schema and this blueprint&#39;s type map, handed in as input.</p><p><strong>The existing DAGs</strong>, for the naming convention to follow and the ids to avoid. With one extra touch — a defect in the repo gets flagged so the model does not copy it:</p><pre>if dag_id != path.name.removesuffix(&quot;.dag.yaml&quot;):<br>    line += &quot;  &lt;-- WARNING: dag_id does not match filename; do not copy this&quot;</pre><p><strong>The connections</strong>, parsed out of airflow/config/airflow_connections.sh — the script that actually creates them. Parsing the provisioning script rather than keeping a second list means the prompt cannot drift from what Airflow really has.</p><p>All four come from one pass, cached for the session. The catalog cannot change mid-conversation, so re-reading it per turn would be slow and pointless.</p><h3>Step 2: the messages</h3><pre>messages = [SystemMessage(content=system)]<br>if history:<br>    messages.extend(history)<br>messages.append(HumanMessage(content=human))</pre><p>The system prompt is static: the grounding rules, a six-step procedure, the output contract, and two worked examples — one success, one refusal. The human message is everything variable: the four grounding blocks, then the request.</p><p>The grounding rules are blunt, and one of them carries most of the weight:</p><blockquote><strong><em>Never invent a config key.</em></strong><em> The config models do not set </em><em>extra=&quot;forbid&quot;, so an invented key is </em><strong><em>silently discarded</em></strong><em> rather than rejected — the DAG parses, runs, and quietly does something other than what the key implies. A made-up </em><em>incremental_column produces a full overwrite that looks like an incremental load. This is the most damaging mistake you can make, precisely because nothing fails.</em></blockquote><p>I will come back to that in step 4, because it is also why the validator exists.</p><p>One detail in the history worth noting. What gets stored per turn is the <strong>raw sentence</strong> and the YAML — not the rendered message with the whole catalog in it:</p><pre>self.history.append(HumanMessage(content=request))<br>self.history.append(AIMessage(content=result.dag.yaml))</pre><p>The grounding block is large. Replaying it for every past turn would send it five times in a five-turn chat and buy nothing, since the newest message already carries the current copy.</p><p>History exists so follow-ups work. “Make it hourly” is not a pipeline request — it only means something next to the previous answer.</p><h3>Step 3: the model call</h3><pre>llm = build_llm(model, provider=provider).with_structured_output(GeneratedDag)</pre><p>That one line changes the contract: invoke() returns a GeneratedDag object rather than a string. We use theGeneratedDag to keep the ouput structured and avoid text-parsing. The LangChain turns the Pydantic model into a JSON schema, hands it to the provider as a tool definition, and the provider constrains generation to match.</p><p>It has a second effect that is easy to miss. The field descriptions become part of the prompt:</p><pre>warnings: list[str] = Field(<br>    default_factory=list,<br>    description=(<br>        &quot;Things that are true and worth knowing even when the DAG is correct: &quot;<br>        &quot;type-mapping gaps, numeric precision loss, PII landing unmasked, &quot;<br>        &quot;destructive overwrite mode, memory pressure from large tables.&quot;<br>    ),<br>)</pre><p>Documentation for the next engineer and instruction for the model, in one place.</p><p>And it makes refusal a typed outcome rather than a special case:</p><pre>class GenerationStatus(str, Enum):<br>    OK = &quot;ok&quot;<br>    NEEDS_CLARIFICATION = &quot;needs_clarification&quot;</pre><p>Every other field defaults to &quot;&quot;, so a refusal is a complete, valid object:</p><pre>{<br>  &quot;status&quot;: &quot;needs_clarification&quot;,<br>  &quot;reasoning&quot;: &quot;No table named `accounts` exists in the source catalog...&quot;,<br>  &quot;yaml&quot;: &quot;&quot;,<br>  &quot;questions&quot;: [<br>    &quot;There is no `accounts` table in the source database. Did you mean `public.users`, which holds account_number, account_type, and balance?&quot;<br>  ]<br>}</pre><p>A model with no way to say “I don’t know” will guess. Giving refusal a status code, a worked example in the prompt, and a code path that treats it as a terminal <em>success</em> is what turns a hallucinated table into a question.</p><p>Three backends sit behind build_llm — the Anthropic API, Bedrock, or a local model via Ollama — selected by LLM_PROVIDER. Temperature is 0: this is code generation against a fixed schema, where the same request producing the same DAG is worth more than variety.</p><h3>Step 4: determenistic validation</h3><p>Grounding is not a guarantee. A grounded model is usually right, and “usually” is not the standard for a file the Airflow scheduler is about to import.</p><p>There is a structural reason to be strict. If the generated file YAML file is malformed- Airflow would fail to import it.</p><p>In order to verify the correctness of the YAML file, it is checked against the live Pydantic model the blueprint will actually be rendered with:</p><pre>model = registry.get(name).get_config_type()</pre><p>Then the check that justifies the whole file:</p><pre>declared = set(model.model_fields)<br>invented = sorted(set(config) - declared)</pre><p>Here is why that set difference has to exist. Pydantic’s default is to <strong>ignore</strong> unknown fields, and these config models do not set extra=&quot;forbid&quot;. So this passes validation cleanly:</p><pre>steps:<br>  ingest_users_table:<br>    blueprint: postgres_to_iceberg_blueprint<br>    table_name: users<br>    incremental_column: created_at     # does not exist<br>    mode: append</pre><p>The YAML parses. The DAG shows up in the Airflow UI. The task group renders. The pipeline runs green every morning — doing a full load, while a line in the file, visible in review, visible in git, says it is incremental.</p><p>That is the worst kind of bug: confidently wrong, with no symptom. model_validate cannot catch it. Only comparing the keys against model_fields can.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-YeVkIJPD8qMtnAP0fBM2w.png" /><figcaption><em>The same file, with and without the check — and why three of the five checks target silence rather than noise.</em></figcaption></figure><p>mode is the same shape of problem by a different route. It is typed as a bare str with no enum, so Pydantic accepts &quot;merge&quot; happily — and the blueprint, which only branches on overwrite and append, silently takes the append path. Checked by hand too.</p><p>Of the five checks in validate.py, <strong>three target silent misbehaviour</strong>: invented keys, bad mode, and start_date that YAML quietly turns into a date object. That ratio is the lesson. Loud failures announce themselves and get fixed. Quiet ones ship and run for months.</p><p>The errors are written to be acted on:</p><pre>f&quot;Step {step_id!r} sets key(s) {invented}, which blueprint {name!r} &quot;<br>f&quot;does not define. These would be silently ignored at runtime, so the &quot;<br>f&quot;DAG would not do what the keys imply. Allowed keys: {sorted(declared)}.&quot;</pre><p>What is wrong, why it matters, and every valid alternative.</p><h3>Step 5: repair, not retry</h3><p>When validation fails, the errors go back to the model in place:</p><pre>validation = validate_dag_yaml(dag.yaml, dag.dag_id, dag.file_path)<br>if validation.ok:<br>    break</pre><pre>if attempt &lt; max_attempts:<br>    messages.append(AIMessage(content=dag.yaml))<br>    messages.append(HumanMessage(content=validation.as_feedback()))</pre><p>The distinction is worth being precise about. A <strong>retry</strong> calls the model again with the same input and throws away everything it got right — the correct table, the right blueprint, the namespace convention, a well-judged schedule. A <strong>repair</strong> hands back the candidate <em>plus the specific reasons it was rejected</em>, and asks for a correction.</p><p>The conversation now reads: rules → request → “here is my DAG” → “these four things are wrong, fix them” → corrected DAG. That is a code review, and models are good at acting on reviews.</p><p>The feedback is addressed to a model, not written for a log file:</p><pre>return (&quot;The YAML you produced failed validation against this repository. &quot;<br>        f&quot;Fix every problem below and return the corrected DAG.\n\n{bullets}&quot;)</pre><p>“Fix every problem below” earns its place — handed four errors without it, a model will often fix the first and return.</p><p>Three attempts by default. The failure curve is steep: a model that cannot fix a named, explicit error in three tries is usually failing for a structural reason — a request the blueprint genuinely cannot express, or a model too weak for the schema.</p><p>This is also what makes weak models safe to try. Point the generator at a local 8B model through <strong><em>Ollama</em></strong> and it will need more attempts than <strong><em>Opus</em></strong>. It will not produce a broken DAG, because the failure mode is a rejected candidate, not a bad file.</p><p>If all three attempts fail, nothing is written. The user gets the errors and the last candidate, so they can see what was nearly right.</p><h3>Nothing is written until you ask</h3><p>Generation never touches disk. In the terminal, save is the only writer. In the browser, POST /api/save is — and it re-checks every condition rather than trusting the button that called it:</p><pre>if result.validation is None or not result.validation.ok:<br>    raise HTTPException(status_code=400,<br>        detail=&quot;The last DAG failed validation; refusing to save it.&quot;)</pre><p>A stale tab is enough to send a request the UI would not offer, and this endpoint writes a file the scheduler imports into a process running every pipeline in the repo. It does not get to trust its caller.</p><h3>The other path: a wizard with no model at all</h3><p>There is a second way in, and it uses no LLM. make wizard asks for one parameter at a time and validates each answer locally — against the registry, the live catalog, the connection list — before moving on.</p><p>The ordering is the design. Integration comes first, so an unsupported pair is refused before a single token would have been spent:</p><pre>Destination system [iceberg]<br>&gt; snowflake<br>  ✗ UNSUPPORTED INTEGRATION: postgres -&gt; snowflake<br>    From postgres, only these destinations exist: iceberg.<br>&gt; iceberg<br>  ✓ iceberg<br>  ✓ blueprint postgres_to_iceberg_blueprint</pre><pre>2. Source  which database, which table<br>Source table<br>&gt; accounts<br>  ✗ No table &#39;accounts&#39; in the source database.<br>    Tables in the source database: public.transactions, public.users.<br>&gt; public.users<br>  ✓ public.users (~100 rows, 5 columns)<br>  ⚠ Possible PII with no masking in the blueprint - these land in Iceberg<br>    in the clear: email, ssn.</pre><p>That refusal is derived, not hardcoded — capabilities.py works out the supported pairs from the registry itself, so a new blueprint adds its pair with no code change.</p><p>The wizard’s most portable idea is that failures come in two kinds, handled differently.</p><p><strong>Hard errors re-ask.</strong> The answer names something that does not exist, so no correct YAML could contain it. A rejected answer never reaches the spec, and since the YAML is rendered only from the spec, an invalid value <em>cannot</em> appear in the output.</p><p><strong>Soft warnings are accepted and carried to the summary.</strong> The pipeline is buildable; these are consequences worth seeing before it runs —</p><ul><li>unmapped types landing as strings</li><li>money losing exactness</li><li>PII unmasked</li><li>no primary key so re-runs cannot deduplicate</li><li>large row counts</li><li>overwrite discarding snapshot history.</li></ul><p>The reasoning is one line in slots.py, and it generalises well past this project:</p><blockquote><em>Blocking on those would stop legitimate pipelines; hiding them would ship a surprise.</em></blockquote><p>A system that blocks on everything questionable gets routed around. <br>A system that warns about nothing ships surprises into production. Separating “this cannot be true” from “this is true and you should know it” is what lets both be strict.</p><p>Both paths end the same way: the summary, every warning collected, the YAML, and a confirmation. And both run the same validator — because a wizard bug is as capable of producing bad YAML as a model is.</p><h3>One last thing: the data boundary</h3><p>For a tool pointed at a production database, this is worth being exact about.</p><p>Only schema metadata reaches the model — table names, column names, types, keys, and a row estimate. No SELECT is ever issued against your tables, so row values never leave the local stack. Column <em>names</em> do go to a hosted provider, which is why ssn and date_of_birth appear in the prompt even though their values never do. With --provider ollama, nothing leaves the machine at all.</p><h3>User interface</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dqbBU-gnKTdUcRvsfCDajA.png" /><figcaption>The chatbot user interface</figcaption></figure><p>The user interface contains a view over available sources and existing dags on the left pannel.</p><p>The chat offers a capability to verbally describe which new sources the users want to integrate as well as example prompts. The interface is available via Web browser and is especially handy for the users without strong technical background.</p><h3>Summary</h3><p>The declarative layer in the previous article moved pipeline definitions from Python into YAML, and split ownership: platform teams own blueprints, data teams own configuration. This layer sits on top of that and removes the archaeology — the blueprint config model, the connections script, the existing DAGs, the psql session — by reading all of it live and putting it in front of a model.</p><p>Four things turned out to matter, and only one of them is the LLM:</p><p><strong>Grounding read live, never hand-maintained.</strong> The registry, the schema, the DAGs and the connections come from the source of truth on every session. A new blueprint is usable immediately because there is no list to update.</p><p><strong>The validator is the product.</strong> The model is a drafting tool; the gate is what makes its output trustworthy. And the checks worth writing are the ones catching <em>silent</em> misbehaviour — an invented key Pydantic drops, a mode value that quietly becomes append. Loud failures mostly announce themselves. Quiet ones ship.</p><p><strong>Refusal is a feature.</strong> needs_clarification and the wizard&#39;s hard errors are the same idea from two directions: a system with no way to say &quot;that does not exist&quot; will invent something instead.</p><p><strong>Hard errors and soft warnings are different.</strong> Re-ask when the answer cannot be true. Warn and carry forward when it is true but costly. Conflating them produces either a tool people bypass or a tool that ships surprises.</p><p>The pattern generalises past Airflow. Any platform with a declarative interface and a registry of capabilities can be fronted this way — introspect what exists, ground the model in it, validate hard against the real models before anything is written, and let the thing refuse. The chatbot is the part people notice. The validator is the part that makes it safe to use.</p><p>Code: <a href="https://github.com/vanikkars/airflow-blueprint-gen-ai">github.com/vanikkars/airflow-blueprint-gen-ai</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=55585acfccea" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Declarative yaml-based airflow pipelines as Data Platform capability]]></title>
            <link>https://medium.com/@vanik_kars/declarative-yaml-based-airflow-pipelines-as-data-platform-capability-e7b3b555e78d?source=rss-42fb2c934fcb------2</link>
            <guid isPermaLink="false">https://medium.com/p/e7b3b555e78d</guid>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[astronomer]]></category>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[apache-iceberg]]></category>
            <dc:creator><![CDATA[Vanik Karslian]]></dc:creator>
            <pubDate>Sun, 20 Sep 2026 20:39:44 GMT</pubDate>
            <atom:updated>2026-10-05T07:32:17.422Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zWbjJ-2IJV0N7K62" /><figcaption>Photo by <a href="https://unsplash.com/@sen7?utm_source=medium&amp;utm_medium=referral">Sen</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>full project in github: <a href="https://github.com/vanikkars/airflow-blue-print-project">https://github.com/vanikkars/airflow-blue-print-project</a></p><h3>Introduction</h3><p>Every data platform team eventually reaches the same point: you have <strong>thousands</strong> <strong>ingestion DAGs</strong>, many working with identical source types and performing the same operations:</p><ol><li><strong>Read the source</strong></li><li><strong>Infer or validate the schema</strong></li><li><strong>Write the data and commit it to a catalog</strong></li></ol><p>Yet each pipeline is a slightly different Python implementation. At enterprise scale, this lack of standardization becomes difficult to maintain: changes that should happen once at the platform level instead require updates across hundreds of DAGs. Which leads to tedious errors and becomes a nightmare to maintain on scale.</p><p>There is also a <strong>usability problem</strong>. Sources such as CSV files and Google Sheets are often owned by business teams without the technical expertise to maintain Python and Airflow code. These people require more user-friendly interface, otherwise the platform risks to not meat the expected adoption rate.</p><p>In this article I will demonstrate how to <strong>standardize</strong> ingestion process via Yaml pipeline definitions and offer it as a platform capability.</p><p>I will use the Astronomer’s Blueprint library <a href="https://lnkd.in/p/esmx2Dbb">shared</a> recently by <a href="https://medium.com/u/64dbef245a10">Marc Lamberti</a> for yaml-to-python translation.</p><p>A little spoiler:</p><pre>---<br>dag_id: banking_users_to_iceberg<br>description: &quot;Extract banking users tables to Iceberg in S3&quot;<br>schedule: &quot;@daily&quot;<br>start_date: &quot;2024-01-01&quot;<br>catchup: false<br><br>steps:<br>  ingest_users_table:<br>    blueprint: postgres_to_iceberg_blueprint<br>    table_name: users<br>    schema_name: public<br>    iceberg_namespace: banking<br>    iceberg_table_name: users<br>    mode: overwrite<br>    postgres_conn_id: postgres_banking</pre><p>That is an entire production-shaped pipeline: Postgres to Apache Iceberg</p><p>The yaml contains the high level configuration:</p><ul><li>name</li><li>description</li><li>schedule — “daily”</li><li>start_date — “2024–01–01&quot;</li><li>whether the catchup is required</li></ul><p>As well as the details for the source and target:</p><ul><li>table_name</li><li>schema_name</li><li>iceberg_namespace</li><li>iceberg_table_name</li></ul><p>All parameters are self-explanatory, to fill in all the values one does not need deep techincal knowledge.</p><p>This abstracaction from deep technical details of the Airflow dag definition creates a shift from: “the platform team writes pipelines” to “the platform team ships a capability, and users invoke it”.</p><p>Such approach simplifies the user experience and creates a clear border between the platform and the data pipeline owner.</p><p>The Python implementation is owned by the platform team. <br>The YAML pipeline definitions are owned by the teams that need the data.</p><h3>Requirements</h3><p>The goals I set for this project were:</p><ol><li>Make the pipeline definition declarative — adding a table should not require writing or reviewing Python</li><li>Offer ingestion as a self-service capability, so non-engineers can create pipelines without a ticket</li><li>Keep the implementation typed and validated — a bad YAML key should fail at DAG-parse time, not at 3am in a task log</li><li>Make the blueprint reusable across any table, without per-table special cases</li><li>Standardise everything the user does not specify — retries, owner, catchup — so consistency is the default rather than a review comment</li><li>Run the entire stack locally, writing to a modern table format so the output is genuinely useful</li></ol><p>Enough talking, let’s jump into the impementation.</p><h3>Implementation</h3><p>The project runs locally in Docker Compose: Airflow 2.11, Postgres for source data and Airflow metadata, a third backing the Iceberg REST catalog, and MinIO as S3-compatible storage. No cloud account required.</p><h4>One blueprint, N pipelines</h4><p>In large enterprises, it is common to operate thousands of ingestion pipelines against homogeneous source types: tables across PostgreSQL or MySQL databases, hundreds of CSV files in S3 buckets, or Google Sheets distributed across corporate drives.</p><p>While the number of pipelines can be large, their execution pattern is often nearly identical: <strong>extract, apply basic transformations, and load</strong>. What typically changes is configuration — the connection, source table or file, destination, and schedule.</p><p>Astronomer Blueprints provide a clean way to centralize this shared logic. The Blueprints library offers it as a Blueprint class.</p><p>Think of the Blueprint as a pipeline <strong>template</strong>: the platform team defines the pipeline’s behavior and the steps it must conduct. <br>For example, the <strong><em>PostgresToIcebergBlueprint</em></strong> defines the complete ingestion workflow — extraction, schema handling, transformation, writing, and catalog commits — in a single Python class.</p><pre>class PostgresToIcebergBlueprint(Blueprint[PostgresToIcebergConfig]):<br>    &quot;&quot;&quot;Blueprint for extracting Postgres tables to Iceberg format in S3.&quot;&quot;&quot;<br><br>    def render(self, config: PostgresToIcebergConfig) -&gt; TaskGroup:<br>        &quot;&quot;&quot;Render the task group for Postgres to Iceberg ingestion.&quot;&quot;&quot;<br><br>        iceberg_table_name = config.iceberg_table_name or config.table_name<br><br>        with TaskGroup(group_id=f&quot;ingest_{config.table_name}&quot;) as task_group:<br><br>            @task(task_id=&quot;extract_postgres_schema&quot;)<br>            def extract_postgres_schema(**context) -&gt; dict[str, Any]:<br>                &quot;&quot;&quot;Extract table schema from Postgres.&quot;&quot;&quot;<br>                ...<br><br>                return {&quot;columns&quot;: columns, &quot;table_name&quot;: config.table_name}<br><br>            @task(task_id=&quot;extract_postgres_data&quot;)<br>            def extract_postgres_data(schema_info: dict[str, Any], **context) -&gt; str:<br>                &quot;&quot;&quot;Extract data from Postgres and save as Parquet.&quot;&quot;&quot;<br>                ...<br><br>                print(f&quot;Extracted {len(df)} rows from {config.table_name}&quot;)<br>                return temp_file.name<br><br>            @task(task_id=&quot;create_iceberg_table&quot;)<br>            def create_iceberg_table(<br>                schema_info: dict[str, Any], **context<br>            ) -&gt; dict[str, str]:<br>                &quot;&quot;&quot;Create or update Iceberg table.&quot;&quot;&quot;<br>               ...<br>                return {<br>                    &quot;table_identifier&quot;: table_identifier,<br>                    &quot;warehouse&quot;: catalog_config[&quot;warehouse&quot;],<br>                }<br><br><br>            @task(task_id=&quot;load_to_iceberg&quot;)<br>            def load_to_iceberg(<br>                parquet_file: str, iceberg_info: dict[str, str], **context<br>            ) -&gt; dict[str, Any]:<br>                &quot;&quot;&quot;Load Parquet data into Iceberg table.&quot;&quot;&quot;<br>                ...<br>                return {<br>                    &quot;table&quot;: iceberg_info[&quot;table_identifier&quot;],<br>                    &quot;rows_loaded&quot;: row_count,<br>                    &quot;mode&quot;: config.mode,<br>                }<br><br><br>            # Define task dependencies<br>            schema_info = extract_postgres_schema()<br>            parquet_file = extract_postgres_data(schema_info)<br>            iceberg_info = create_iceberg_table(schema_info)<br>            result = load_to_iceberg(parquet_file, iceberg_info)<br><br>        return task_group</pre><p>The project ships one blueprint implementation — postgres_to_iceberg.py, roughly 300 lines.</p><p>Now, we can define two piplines that would share the common logic, defined in PostgresToIcebergBlueprint:</p><p>the one ingests the <em>transactions</em>:</p><pre>steps:<br>  ingest_transactions_table:<br>    blueprint: postgres_to_iceberg_blueprint<br>    table_name: transactions<br>    schema_name: public<br>    iceberg_namespace: banking<br>    iceberg_table_name: transactions<br>    s3_bucket: iceberg-warehouse<br>    s3_prefix: warehouse<br>    mode: overwrite<br>    batch_size: 100<br>    postgres_conn_id: postgres_banking</pre><p>the other — the <em>users</em> table:</p><pre>steps:<br>  ingest_users_table:<br>    blueprint: postgres_to_iceberg_blueprint<br>    table_name: users<br>    schema_name: public<br>    iceberg_namespace: banking<br>    iceberg_table_name: users<br>    s3_bucket: iceberg-warehouse<br>    s3_prefix: warehouse<br>    mode: overwrite<br>    batch_size: 100<br>    postgres_conn_id: postgres_banking</pre><p>As you can see the creation of a new pipeline with users just required to change the input and output parameters, and that’s it.</p><p>Everything else — the schema extraction, the Postgres-to-Iceberg type mapping, the batched read, the Parquet staging, the catalog commit, the timestamp fixes, the retry policy — is inherited. Not copied. <strong>Inherited</strong>.</p><p>Now, in order to integrate another source table into iceberg destination one need to create a new yaml file.</p><p>This is the difference between a template and a capability:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s5lKGIy-SRSDIvwofClBGg.png" /></figure><h4>The YAML file is the interface</h4><p>If the blueprint is the capability developed by the data platform team and offered to the end customers. Thus, the YAML file plays a role of a public inteface. Given that it’s a public intrface we should also validate the user input. Here is when the Pydantic model plays a crucial role. The pydantic allows provides us the validation out of the box- which fields are required, what are the expected values and types:</p><pre>class PostgresToIcebergConfig(BaseModel):<br>    &quot;&quot;&quot;Configuration for Postgres to Iceberg ingestion.&quot;&quot;&quot;<br><br>    table_name: str = Field(..., description=&quot;Name of the Postgres table to ingest&quot;)<br>    schema_name: str = Field(default=&quot;public&quot;, description=&quot;Postgres schema name&quot;)<br>    iceberg_namespace: str = Field(<br>        default=&quot;default&quot;, description=&quot;Iceberg namespace/database&quot;<br>    )<br>    mode: str = Field(default=&quot;overwrite&quot;, description=&quot;Write mode: &#39;overwrite&#39; or &#39;append&#39;&quot;)<br>    batch_size: int = Field(default=10000, description=&quot;Number of rows to fetch per batch&quot;)<br>    postgres_conn_id: str = Field(<br>        default=&quot;postgres_banking&quot;, description=&quot;Airflow connection ID for source Postgres&quot;<br>    )</pre><p>in order to invoke the blue-print, one need to merely define it’s name in the pipeline’s yaml:</p><pre>steps:<br>  ingest_loans:<br>    blueprint: postgres_to_iceberg_blueprint<br>    table_name: loans</pre><p>As a bonus of using Pydantic — each yaml file gets validated before anything runs. Pydantic checks the config at DAG-parse time, thus any mistakes or typos in the yaml file, such as iceberg_names<strong>pcae</strong> will be captured immediately.</p><p>The blast radius is bounded. A YAML author chooses <em>which</em> table, <em>where</em> it lands, and <em>how often </em>and the validation comes out of the box<em>.</em></p><p>The ownership split ends up clean:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FehCAXilfy3u1WSHWGQXdA.png" /></figure><p>essentially, the platofrm team owns the pipeline’s implementation details and the users — their pipelines defined in yaml.</p><h4>Standardisation and code duplicaiton</h4><p>Let’s now see how the blue prints help to avoid code duplication. Imagine, we want to provide reasonable default values for certain pipeline parameters, such as number of retries or retry delay. In order to implement it, we need to define the <em>ProjectDagArgsConfig </em>class<em>:</em></p><pre>class ProjectDagArgsConfig(BaseModel):<br>    schedule: str | None = None<br>    start_date: str | None = None<br>    catchup: bool = False<br>    owner: str = &quot;data-engineering&quot;<br>    retries: int = 2<br>    retry_delay_minutes: int = 5<br><br><br>class ProjectDagArgs(BlueprintDagArgs[ProjectDagArgsConfig], default=True):<br>    &quot;&quot;&quot;DAG arguments shared by every pipeline in this project.&quot;&quot;&quot;<br><br>    def render(self, config: ProjectDagArgsConfig) -&gt; dict[str, Any]:<br>        return {<br>            &quot;catchup&quot;: config.catchup,<br>            &quot;default_args&quot;: {<br>                &quot;owner&quot;: config.owner,<br>                &quot;retries&quot;: config.retries,<br>                &quot;retry_delay&quot;: timedelta(minutes=config.retry_delay_minutes),<br>            },<br>            ...<br>        }</pre><p>default=True applies this to every DAG in the project. Every pipeline retries twice, carries an owner, and has catchup off — not because each author remembered, but because those are the defaults. Nobody writing YAML needs to know what catchup does to avoid being burned by it.</p><p>Three consequences: platform-wide policy becomes a one-line edit rather than forty. The schema acts as a whitelist — users cannot set an arbitrary Airflow parameter, so they cannot set a harmful one. And variance disappears: templating removes duplication, this removes <em>inconsistency</em>, and only the second is what people mean by “standardised pipelines.”</p><h4>Rendering the task group</h4><p>A Blueprint is the config model above plus a render() method turning it into tasks. It returns a TaskGroup, so each YAML step becomes a grouped unit in the Airflow UI:</p><pre>class PostgresToIcebergBlueprint(Blueprint[PostgresToIcebergConfig]):<br>    &quot;&quot;&quot;Blueprint for extracting Postgres tables to Iceberg format in S3.&quot;&quot;&quot;<br><br>    def render(self, config: PostgresToIcebergConfig) -&gt; TaskGroup:<br>        iceberg_table_name = config.iceberg_table_name or config.table_name<br><br>        with TaskGroup(group_id=f&quot;ingest_{config.table_name}&quot;) as task_group:<br><br>            @task(task_id=&quot;extract_postgres_schema&quot;)<br>            def extract_postgres_schema(**context) -&gt; dict[str, Any]: ...<br>            # extract_postgres_data / create_iceberg_table / load_to_iceberg<br>            # follow the same shape, each reading `config` from the closure<br><br>            schema_info = extract_postgres_schema()<br>            parquet_file = extract_postgres_data(schema_info)<br>            iceberg_info = create_iceberg_table(schema_info)<br>            result = load_to_iceberg(parquet_file, iceberg_info)<br><br>        return task_group</pre><p>config is captured by closure, so each task reads config.table_name or config.mode directly — no XCom plumbing for configuration, no params dictionary to get wrong.</p><h4>The loader: how YAML becomes DAGs</h4><p>Airflow only executes Python files; it will never turn a bare .yaml into a DAG. One twelve-line module bridges the gap, and is never touched again:</p><pre>build_all_airflow_dags(<br>    search_path=_DAGS_DIR,<br>    register_globals=globals(),<br>    bp_registry=BlueprintRegistry(template_dirs=[_BLUEPRINTS_DIR, _DAGS_DIR]),<br>)</pre><p>It discovers every *.dag.yaml, resolves each step&#39;s blueprint: key against the registry, validates the config against the Pydantic model, and injects the DAG objects into module globals for the scheduler.</p><p>This is what makes the whole thing self-service: a new pipeline requires no registration step. Drop a file into dags/, and it exists — no list to append to, nothing a new user can forget.</p><h4>The unglamorous fixes that reusability pays for</h4><p>Two type problems bit me, and they are the clearest argument for centralising an implementation.</p><ol><li><em>Nanosecond timestamps. <br></em>pandas produces timestamp[ns]; Iceberg tops out at microseconds, so an untouched frame fails with UnsupportedPyArrowTypeException:</li></ol><pre>def _downcast_ns_timestamps(table: &quot;pa.Table&quot;) -&gt; &quot;pa.Table&quot;:<br>    &quot;&quot;&quot;Cast any nanosecond timestamp column to microseconds.&quot;&quot;&quot;<br>    import pyarrow as pa<br><br>    fields = []<br>    changed = False<br>    for field in table.schema:<br>        if pa.types.is_timestamp(field.type) and field.type.unit == &quot;ns&quot;:<br>            fields.append(field.with_type(pa.timestamp(&quot;us&quot;, tz=field.type.tz)))<br>            changed = True<br>        else:<br>            fields.append(field)<br><br>    if not changed:<br>        return table<br>    return table.cast(pa.schema(fields))</pre><p>2. <em>Schema drift. </em><br>pandas widens ints to int64 and marks every column nullable, which Iceberg rejects for int32 or NOT NULL columns. The fix is one line — arrow_table.cast(table.schema().as_arrow()) — casting to the target table&#39;s own Arrow schema before writing.</p><p>Neither fix is interesting, and that is the point. Nobody writing YAML will encounter either, or need to learn the words “PyArrow schema coercion.” The platform absorbed both, permanently, for every pipeline that will ever exist.</p><p>This is the real argument for blueprints. Not that YAML is prettier than Python — that a hard-won fix has exactly one home, and the people creating pipelines never have to learn the problem it solves.</p><h3><strong>Summary</strong></h3><p>Let’s some up what has been implemented.</p><ol><li>We have created a Blueprint class <strong><em>PostgresToIcebergBlueprint</em></strong> that extracts data from postgres and ingests into Iceberg</li><li>We defined <strong><em>ProjectDagArgsConfig</em></strong> to store the default params for all the dags in the project</li><li>We provided a yaml-based public interface for our users</li></ol><p>And lust but not least, we introduce a significant change in the approach- a clear separation between platform and users ownership:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ju_Bvlo250CgFAnTXeaN9g.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e7b3b555e78d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Data Contract Management System with AWS Glue Schema Registry]]></title>
            <link>https://medium.com/@vanik_kars/building-a-data-contract-management-system-with-aws-glue-schema-registry-4f883d1b97f9?source=rss-42fb2c934fcb------2</link>
            <guid isPermaLink="false">https://medium.com/p/4f883d1b97f9</guid>
            <category><![CDATA[kafka-schema-registry]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-contract]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-engineer]]></category>
            <dc:creator><![CDATA[Vanik Karslian]]></dc:creator>
            <pubDate>Tue, 11 Aug 2026 07:01:12 GMT</pubDate>
            <atom:updated>2026-08-11T19:30:35.160Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Z5HG7ULDNfr7Sfjo" /><figcaption>Photo by <a href="https://unsplash.com/@markuswinkler?utm_source=medium&amp;utm_medium=referral">Markus Winkler</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>full project in github: <a href="https://github.com/vanikkars/schema-registry">https://github.com/vanikkars/flink-ingestion</a></p><h3>Introduction</h3><p>In recent years, I’ve seen data contracts become a requirement in many modern organizations. They tackle a super important aspect of working with data: <strong>the</strong> <strong>trust</strong> between the source and the producer.</p><p>Although the industry has accepted that we need some sort of agreement between data producers and consumers, many companies still struggle to incorporate data contracts into their data toolbox — especially when their codebase contains lots of legacy systems with various data sources.</p><p>In this article, we demonstrate a data contract management system designed to address this need using <strong>AWS Glue Schema Registry.</strong></p><h3>Requirements</h3><p>As key requirements, we would like to:</p><ol><li>Provide a source-agnostic data contract</li><li>Provide a working solution as soon as possible</li><li>Avoid reinventing the wheel — pragmatically use existing cloud solutions when they fit</li><li>Reduce the maintenance burden on the platform team</li><li>Ensure the solution integrates natively into the day-to-day development flow of both producers and consumers</li></ol><p>With that, let’s jump into the implementation.</p><h3>Implementation</h3><h4>Architecture Decisions</h4><h4>The Schema Registry</h4><p>In this article, I’ll introduce you to Schema Registry — a centralized registry for all data contracts in the company. In this example, we’ll use the registry provided by AWS as our core. Here are the reasons why:</p><ul><li><strong>Managed serverless solution</strong>: No headache for maintenance</li><li><strong>Complex compatibility modes</strong>: Backward, backward_all, forward, forward_all, and full compatibility checking — we want to avoid reinventing the wheel</li><li><strong>Modern format support</strong>: Natively supports AVRO, JSON, and Protobuf</li><li><strong>Version management</strong>: Keeps all versions ever ingested into the registry</li><li><strong>Cost</strong>: It’s a free service in AWS</li></ul><h4>Automation</h4><p>We also want to automate the provisioning of resources. For example, when a schema is updated (a new column is added), we want to verify if the change is compatible and, if so, update the corresponding table in Iceberg.</p><p>We also want to incorporate the solution into the native development flow for developers. Thus, we’ll include verification steps as part of the CI/CD process using GitHub Actions, which will conduct the required checks and provide corresponding feedback to the user.</p><h4>Contract Format</h4><p>For the contract format, I propose using a source-agnostic format based on a simplified version of the Open Data Contract Standard. Contracts themselves will be stored in JSON format, which is source-agnostic and very human-friendly — especially handy during code reviews.</p><p>Each contract contains the schema definition as well as metadata columns, such as SLAs and the owner:</p><pre>{<br>  &quot;contract_id&quot;: &quot;users-v1&quot;,<br>  &quot;name&quot;: &quot;Users&quot;,<br>  &quot;description&quot;: &quot;Schema for user records&quot;,<br>  &quot;version&quot;: &quot;1.0.0&quot;,<br>  &quot;columns&quot;: [<br>    {<br>      &quot;name&quot;: &quot;user_name&quot;,<br>      &quot;data_type&quot;: &quot;string&quot;,<br>      &quot;nullable&quot;: false,<br>      &quot;description&quot;: &quot;Username for the user&quot;<br>    },<br>    {<br>      &quot;name&quot;: &quot;email&quot;,<br>      &quot;data_type&quot;: &quot;string&quot;,<br>      &quot;nullable&quot;: false,<br>      &quot;description&quot;: &quot;Email address of the user&quot;<br>    },<br>    {<br>      &quot;name&quot;: &quot;date_of_birth&quot;,<br>      &quot;data_type&quot;: &quot;date&quot;,<br>      &quot;nullable&quot;: false,<br>      &quot;description&quot;: &quot;Date of birth in YYYY-MM-DD format&quot;<br>    }<br>  ],<br>  &quot;metadata&quot;: {<br>    &quot;data_owner&quot;: &quot;User Management&quot;,<br>    &quot;data_owner_email&quot;: &quot;user-mgmt@company.com&quot;,<br>    &quot;data_steward&quot;: &quot;Data Engineering&quot;,<br>    &quot;data_steward_email&quot;: &quot;data-engineering@company.com&quot;,<br>    &quot;sla_uptime_percentage&quot;: 99.95,<br>    &quot;sla_max_latency_ms&quot;: 5000<br>  }<br>}</pre><h4>Registry API</h4><p>To provide a native REST API interface for our GitHub Actions, we built a lightweight proxy service that accepts requests to create/update data contracts, conducts necessary validations, and returns a response.</p><h4>Iceberg Management Service</h4><p>For every data entity that needs to be created, we propose leveraging a dedicated lightweight API service that hides the complexity of infrastructure management and provides a user-friendly REST API interface for the required services (GitHub Actions in this case).</p><h3>High-Level Architecture</h3><p>To address the requirements, I propose keeping contracts as close to the source as possible — in the producer’s repository.</p><p>Once contracts are updated and a PR is published, GitHub Actions will trigger two major steps:</p><ol><li>Verify if the contract is compatible with the previous version</li><li>If compatibility is successful, alter the Iceberg table definition</li></ol><p>Once both steps are completed, a message will be published to the PR.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o6NhvUWwIG5IKjNA0Bp9-g.png" /></figure><h3>Showcase</h3><p>Let’s showcase a few example flows of the entire system end-to-end.</p><h4>1. Creating a New Contract</h4><p>Let’s imagine we’ve prepared a new User data product and are ready to make it live. Here’s how the data contract looks:</p><pre>{<br>  &quot;contract_id&quot;: &quot;users-v1&quot;,<br>  &quot;name&quot;: &quot;Users&quot;,<br>  &quot;description&quot;: &quot;Schema for user records&quot;,<br>  &quot;version&quot;: &quot;1.0.0&quot;,<br>  &quot;columns&quot;: [<br>    {<br>      &quot;name&quot;: &quot;user_name&quot;,<br>      &quot;data_type&quot;: &quot;string&quot;,<br>      &quot;nullable&quot;: false,<br>      &quot;description&quot;: &quot;Username for the user&quot;<br>    },<br>    {<br>      &quot;name&quot;: &quot;email&quot;,<br>      &quot;data_type&quot;: &quot;string&quot;,<br>      &quot;nullable&quot;: false,<br>      &quot;description&quot;: &quot;Email address of the user&quot;<br>    },<br>    {<br>      &quot;name&quot;: &quot;date_of_birth&quot;,<br>      &quot;data_type&quot;: &quot;date&quot;,<br>      &quot;nullable&quot;: false,<br>      &quot;description&quot;: &quot;Date of birth in YYYY-MM-DD format&quot;<br>    }<br>  ],<br>  &quot;metadata&quot;: {<br>    &quot;data_owner&quot;: &quot;User Management&quot;,<br>    &quot;data_owner_email&quot;: &quot;user-mgmt@company.com&quot;,<br>    &quot;data_steward&quot;: &quot;Data Engineering&quot;,<br>    &quot;data_steward_email&quot;: &quot;data-engineering@company.com&quot;,<br>    &quot;sla_uptime_percentage&quot;: 99.95,<br>    &quot;sla_max_latency_ms&quot;: 5000<br>  }<br>}</pre><p>This contract contains three fields:</p><ul><li><strong>user_name</strong> (string, not null)</li><li><strong>email</strong> (string, not null)</li><li><strong>date_of_birth</strong> (date, not null)</li></ul><p>We put the contract file user_v1.json inside the contracts/current/user/ folder and submit our first Pull Request: <a href="https://github.com/vanikkars/schema-registry/pull/32">https://github.com/vanikkars/schema-registry/pull/3</a>1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/1*FfIcYrwmZRk25Z7cWO8amw.png" /><figcaption>Github actions in our first pull request with the new contract.</figcaption></figure><p>Once all checks pass, we can go to the AWS account and check the generated resources:</p><ol><li>We can see that a new schema has been added to the Schema Registry:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*44s21CFhekJWWTOyF7uKIg.png" /><figcaption>New schema in the AWS GLue Schema Registry</figcaption></figure><p>2. We can verify that the defined columns correspond to the fields defined in our contract:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xuoEVT1shT1vOR-YiPGkEA.png" /><figcaption>Schema Details in the AWS GLue Schema Registry</figcaption></figure><p>3. We can navigate to the Glue Catalog and check the Iceberg tables — we see that the table was successfully created</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ELj_cUEdP_Xn2U9GXVZhig.png" /><figcaption>Iceberg table in AWS GLue Catalog with contract 0.0.1</figcaption></figure><h4>2. Updating an Existing Contract</h4><p>Let’s assume we wanted to extend the existing contract by adding two optional fields:</p><ul><li><strong>zip_code</strong> (string, nullable)</li><li><strong>last_login_at</strong> (timestamp, nullable)</li></ul><p>We update the contract user_v1.json inside contracts/current/user/:</p><pre>{<br>  &quot;contract_id&quot;: &quot;users-v1&quot;,<br>  &quot;name&quot;: &quot;Users&quot;,<br>  &quot;description&quot;: &quot;Schema for user records (v2 with profile fields)&quot;,<br>  &quot;version&quot;: &quot;2.0.0&quot;,<br>  &quot;columns&quot;: [<br>    {<br>      &quot;name&quot;: &quot;user_name&quot;,<br>      &quot;data_type&quot;: &quot;string&quot;,<br>      &quot;nullable&quot;: false,<br>      &quot;description&quot;: &quot;Username for the user&quot;<br>    },<br>    {<br>      &quot;name&quot;: &quot;email&quot;,<br>      &quot;data_type&quot;: &quot;string&quot;,<br>      &quot;nullable&quot;: false,<br>      &quot;description&quot;: &quot;Email address of the user&quot;<br>    },<br>    {<br>      &quot;name&quot;: &quot;date_of_birth&quot;,<br>      &quot;data_type&quot;: &quot;date&quot;,<br>      &quot;nullable&quot;: false,<br>      &quot;description&quot;: &quot;Date of birth in YYYY-MM-DD format&quot;<br>    },<br>    {<br>      &quot;name&quot;: &quot;zip_code&quot;,<br>      &quot;data_type&quot;: &quot;string&quot;,<br>      &quot;nullable&quot;: true,<br>      &quot;description&quot;: &quot;Zip code of the user (added in v2, optional for backwards compatibility)&quot;<br>    },<br>    {<br>      &quot;name&quot;: &quot;last_login_at&quot;,<br>      &quot;data_type&quot;: &quot;timestamp&quot;,<br>      &quot;nullable&quot;: true,<br>      &quot;description&quot;: &quot;Timestamp of last login (added in v2, optional for backwards compatibility)&quot;<br>    }<br>  ],<br>  &quot;metadata&quot;: {<br>    &quot;data_owner&quot;: &quot;User Management&quot;,<br>    &quot;data_owner_email&quot;: &quot;user-mgmt@company.com&quot;,<br>    &quot;data_steward&quot;: &quot;Data Engineering&quot;,<br>    &quot;data_steward_email&quot;: &quot;data-engineering@company.com&quot;,<br>    &quot;sla_uptime_percentage&quot;: 99.95,<br>    &quot;sla_max_latency_ms&quot;: 5000<br>  }<br>}</pre><p>We create a new Pull Request: <a href="https://github.com/vanikkars/schema-registry/pull/32">https://github.com/vanikkars/schema-registry/pull/32</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/927/1*ICJIv9_rRkxbD9wby7dPDw.png" /><figcaption>Second PR, added 2 new columns</figcaption></figure><p>Once the PR is merged, we can navigate to the AWS account again. The schema version in the Schema Registry has been incremented from 1 to 2, and the new fields have been successfully added to the schema. We can also verify in the Glue Catalog that the Iceberg table was successfully updated.</p><p>Schema Registry:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OYlVHJqAa2diEoqVLEZ-4g.png" /><figcaption>Two new columns were successfully added to schema (AWS Glue schema registry)</figcaption></figure><p>Iceberg table in Glue catalog now have 2 additional columns:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jPRp4CSZduGth5npY4UX8w.png" /><figcaption>Iceberg table in Glue catalog</figcaption></figure><h4>3. Attempting a Non-Compatible Change</h4><p>Let’s verify the unhappy path. Let’s assume we introduce a non-compatible change into the schema by removing user_name, which is a required field:</p><pre>{<br>  &quot;contract_id&quot;: &quot;users-v1&quot;,<br>  &quot;name&quot;: &quot;Users&quot;,<br>  &quot;description&quot;: &quot;Schema for user records&quot;,<br>  &quot;version&quot;: &quot;3.0.0&quot;,<br>  &quot;columns&quot;: [<br>    {<br>      &quot;name&quot;: &quot;email&quot;,<br>      &quot;data_type&quot;: &quot;string&quot;,<br>      &quot;nullable&quot;: false,<br>      &quot;description&quot;: &quot;Email address of the user&quot;<br>    },<br>    {<br>      &quot;name&quot;: &quot;date_of_birth&quot;,<br>      &quot;data_type&quot;: &quot;date&quot;,<br>      &quot;nullable&quot;: false,<br>      &quot;description&quot;: &quot;Date of birth in YYYY-MM-DD format&quot;<br>    },<br>    {<br>      &quot;name&quot;: &quot;zip_code&quot;,<br>      &quot;data_type&quot;: &quot;string&quot;,<br>      &quot;nullable&quot;: true,<br>      &quot;description&quot;: &quot;Zip code of the user&quot;<br>    },<br>    {<br>      &quot;name&quot;: &quot;last_login_at&quot;,<br>      &quot;data_type&quot;: &quot;timestamp&quot;,<br>      &quot;nullable&quot;: true,<br>      &quot;description&quot;: &quot;Timestamp of last login&quot;<br>    }<br>  ],<br>  &quot;metadata&quot;: {<br>    &quot;data_owner&quot;: &quot;User Management&quot;,<br>    &quot;data_owner_email&quot;: &quot;user-mgmt@company.com&quot;,<br>    &quot;data_steward&quot;: &quot;Data Engineering&quot;,<br>    &quot;data_steward_email&quot;: &quot;data-engineering@company.com&quot;,<br>    &quot;sla_uptime_percentage&quot;: 99.95,<br>    &quot;sla_max_latency_ms&quot;: 5000<br>  }<br>}</pre><p>When we create a pull request, GitHub Actions will trigger validation requests to the Registry API service. The Registry API will detect non-compatible changes and block the further steps. An error message with details will be published to the PR (<a href="https://github.com/vanikkars/schema-registry/pull/32">https://github.com/vanikkars/schema-registry/pull/3</a>4) , preventing the merge until the issue is resolved.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/888/1*3zEGsHVfA7G1LWa7bWzQOw.png" /><figcaption>Github Actions (non-compatible change detected)</figcaption></figure><h3>Deep Dive: Building the System</h3><p>Now that we’ve seen the system in action, let’s explore how each component works under the hood.</p><h3>Service 1: Registry API — The Schema Management Backbone</h3><p>The Registry API is the heart of the system. It’s a FastAPI service that manages data contract registration, versioning, and metadata.</p><h3>Design Pattern: Hexagonal Architecture (Ports &amp; Adapters)</h3><p>The Registry API follows hexagonal architecture principles, which keep business logic decoupled from framework details:</p><pre># app/main.py - Application Composition Root<br>def create_app() -&gt; FastAPI:<br>    # Instantiate outbound adapters (AWS Glue)<br>    schema_registry_adapter = GlueSchemaRegistryAdapter()<br>    <br>    # Instantiate use cases with dependency injection<br>    register_schema_use_case = RegisterSchemaUseCase(<br>        schema_registry=schema_registry_adapter<br>    )<br>    list_schemas_use_case = ListSchemasUseCase(<br>        schema_registry=schema_registry_adapter<br>    )<br>    # ... more use cases<br>    <br>    # Mount router with injected dependencies<br>    router = create_router(<br>        register_schema_use_case=register_schema_use_case,<br>        list_schemas_use_case=list_schemas_use_case,<br>        # ...<br>    )<br>    app.include_router(router)<br>    return app</pre><h3>Directory Structure: Clean Separation of Concerns</h3><pre>registry_api/<br>├── domain/            # Business logic &amp; entities<br>│   ├── models.py      # DataContract, SchemaVersion, Metadata<br>│   └── exceptions.py  # DomainException, SchemaNotFound, etc.<br>│<br>├── application/       # Use cases &amp; orchestration<br>│   └── use_cases.py   # RegisterSchema, ListSchemas, GetSchema, etc.<br>│<br>├── adapters/<br>│   ├── inbound/       # HTTP API<br>│   │   └── http/<br>│   │       └── router.py<br>│   └── outbound/      # External services<br>│       └── aws_glue/<br>│           └── schema_registry_adapter.py<br>│<br>├── app/<br>│   └── main.py        # FastAPI application factory<br>│<br>└── requirements.txt   # Dependencies</pre><h3>Key Responsibilities</h3><p><strong>Domain Layer</strong> (domain/models.py):</p><ul><li>Defines core business entities: DataContract, SchemaVersion, Metadata</li><li>Contains business rules and validations</li><li>No dependencies on external frameworks</li></ul><p><strong>Application Layer</strong> (application/use_cases.py):</p><ul><li>Implements business logic as use cases: RegisterSchemaUseCase, ListSchemasUseCase, etc.</li><li>Orchestrates interactions between domain and adapters</li><li>Keeps business logic framework-agnostic</li></ul><p><strong>Adapter Layer</strong> (adapters/):</p><ul><li><strong>Inbound</strong>: HTTP router that translates HTTP requests to use cases</li><li><strong>Outbound</strong>: GlueSchemaRegistryAdapter that translates use cases to AWS API calls</li></ul><h3>API Endpoints</h3><p>The Registry API exposes RESTful endpoints following OpenAPI standards:</p><pre># Health Check<br>GET /health<br>GET /</pre><pre># Schema Management<br>POST   /api/v1/schemas                          # Register schema<br>GET    /api/v1/schemas                          # List all schemas<br>GET    /api/v1/schemas/{schema_name}            # Get latest schema<br>GET    /api/v1/schemas/{schema_name}/versions   # List all versions<br>GET    /api/v1/schemas/{schema_name}/versions/{v}  # Get specific version<br>DELETE /api/v1/schemas/{schema_name}            # Delete schema</pre><h3>Service 2: Iceberg Creation Service — Table Materialization</h3><p>While the Registry API manages schema definitions, the Iceberg Creation Service materializes those schemas into actual data tables using Apache Iceberg format in AWS Glue.</p><h3>Why Apache Iceberg?</h3><p>Apache Iceberg provides:</p><ul><li><strong>ACID transactions</strong>: Reliable data mutations</li><li><strong>Schema evolution</strong>: Safe schema changes without rewriting data</li><li><strong>Time travel</strong>: Query historical versions of data</li><li><strong>Partition evolution</strong>: Change partitioning without data reorganization</li></ul><h3>Iceberg Service API</h3><pre>POST   /api/v1/tables                  # Create Iceberg table<br>POST   /api/v1/tables/{table_name}/schema  # Update table schema<br>GET    /api/v1/tables/{table_name}     # Get table information<br>GET    /health                         # Health check</pre><h3>Request Example</h3><pre>{<br>  &quot;contract_id&quot;: &quot;users_v1&quot;,<br>  &quot;name&quot;: &quot;users&quot;,<br>  &quot;version&quot;: 1,<br>  &quot;columns&quot;: [<br>    {&quot;name&quot;: &quot;id&quot;, &quot;data_type&quot;: &quot;string&quot;},<br>    {&quot;name&quot;: &quot;user_name&quot;, &quot;data_type&quot;: &quot;string&quot;},<br>    {&quot;name&quot;: &quot;email&quot;, &quot;data_type&quot;: &quot;string&quot;}<br>  ]<br>}</pre><h3>Design: Mirroring the Registry API</h3><p>The Iceberg Service uses the same architectural principles as the Registry API:</p><pre>Iceberg Service Architecture<br>    ├── domain/            # Business logic for table operations<br>    ├── application/       # Use cases for table creation/updates<br>    ├── adapters/<br>    │   ├── inbound/       # HTTP API<br>    │   └── outbound/      # AWS Glue integration<br>    └── app/main.py        # FastAPI application</pre><h3>Service 3: GitHub Actions Automation — The Workflow Engine</h3><p>The magic happens in GitHub Actions. When contracts change, automated workflows:</p><ol><li>Validate contract structure and compatibility</li><li>Register schemas in the Registry API</li><li>Create Iceberg tables</li><li>Post validation results to pull requests</li><li>Auto-merge valid contracts</li></ol><h3>Workflow: Contract Validation Automation</h3><pre>name: Contract Validation Workflow<br>on:<br>  pull_request:<br>    paths:<br>      - &#39;contracts/current/**&#39;</pre><pre>jobs:<br>  validate:<br>    runs-on: ubuntu-latest<br>    steps:<br>      - uses: actions/checkout@v4<br>      <br>      - name: Validate Contracts<br>        run: |<br>          python scripts/validate-contracts.py</pre><pre>- name: Register Schemas<br>        run: |<br>          curl -X POST &quot;https://api.schema-registry.dev/api/v1/schemas&quot; \<br>            -H &quot;Content-Type: application/json&quot; \<br>            -d @contracts/current/user/user_v1.json<br>      <br>      - name: Create Iceberg Tables<br>        run: |<br>          curl -X POST &quot;https://iceberg.schema-registry.dev/api/v1/tables&quot; \<br>            -H &quot;Content-Type: application/json&quot; \<br>            -d @create-table-request.json<br>      <br>      - name: Post Results to PR<br>        if: always()<br>        uses: actions/github-script@v7<br>        with:<br>          script: |<br>            github.rest.issues.createComment({<br>              issue_number: context.issue.number,<br>              owner: context.repo.owner,<br>              repo: context.repo.repo,<br>              body: &#39;✅ Contract validation passed!&#39;<br>            })</pre><h3>Key Design Decisions</h3><ol><li><strong>Only </strong>contracts/current/<strong> triggers workflows</strong>: Production-ready contracts only</li><li><strong>Archive in </strong>contracts/all/: Historical versions for reference</li><li><strong>Automatic merge</strong>: Valid contracts auto-merge without manual review</li><li><strong>Public URLs via Bore</strong>: Expose local services to GitHub Actions without VPN</li></ol><h3>Folder Structure: Contracts Organization</h3><p>Understanding how contracts are organized is crucial:</p><pre>contracts/<br>├── current/                   # ✅ Triggers GitHub Actions<br>│   ├── user/<br>│   │   └── user_v1.json      # Latest, active contract<br>│   └── transaction/<br>│       └── transaction_v1.json<br>│<br>└── all/                       # ❌ Archive (not validated by Actions)<br>    ├── user/<br>    │   ├── 01/user_v1.json   # Version 1<br>    │   ├── 02/user_v1.json   # Version 2<br>    │   └── 03/user_v1.json   # Version 3<br>    └── transaction/<br>        ├── 01/transaction_v1.json<br>        └── 02/transaction_v1.json</pre><h3>Workflow Example: Updating a Contract</h3><ol><li><strong>Developer edits</strong> contracts/current/user/user_v1.json</li><li><strong>Commits and pushes</strong> to feature branch</li><li><strong>Creates pull request</strong> on GitHub</li><li><strong>GitHub Actions triggers</strong>: Validates the contract</li><li><strong>Validation passes</strong>: Action posts success comment</li><li><strong>Auto-merge activates</strong>: Merges PR automatically</li><li><strong>Archive manually</strong>: Copy validated version to contracts/all/user/03/</li></ol><h3>Infrastructure: Docker Compose Setup</h3><p>For local development, both services run in Docker:</p><pre>version: &#39;3.8&#39;<br>services:<br>  registry-api:<br>    build: ./registry_api<br>    ports:<br>      - &quot;8000:8000&quot;<br>    environment:<br>      - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}<br>      - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}<br>      - AWS_DEFAULT_REGION=us-east-1</pre><pre>  iceberg-creation:<br>    build: ./iceberg_creation_service<br>    ports:<br>      - &quot;8001:8001&quot;<br>    environment:<br>      - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}<br>      - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}<br>      - AWS_DEFAULT_REGION=us-east-1</pre><h3>Exposing Services to GitHub Actions</h3><p>For GitHub Actions to validate contracts in PRs, local services need public URLs. The project uses <strong>Bore</strong>, a minimal tunneling solution:</p><pre>brew install bore-cli<br>docker-compose up -d<br>make bore-start  # Exposes services with public URLs</pre><p>This creates tunnels without requiring additional setup or authentication, making it perfect for development workflows.</p><h3>AWS Glue Integration: The Persistence Layer</h3><p>AWS Glue Schema Registry serves as the persistence layer for all schema definitions:</p><pre># adapters/outbound/aws_glue/schema_registry_adapter.py</pre><pre>class GlueSchemaRegistryAdapter:<br>    def __init__(self):<br>        self.client = boto3.client(&#39;glue&#39;)<br>    <br>    def register_schema(self, contract: DataContract) -&gt; SchemaVersion:<br>        &quot;&quot;&quot;Register schema in AWS Glue Schema Registry.&quot;&quot;&quot;<br>        response = self.client.create_schema(<br>            RegistryId={&#39;RegistryName&#39;: &#39;schema-registry&#39;},<br>            SchemaName=contract.name,<br>            DataFormat=&#39;AVRO&#39;,<br>            Compatibility=&#39;BACKWARD&#39;,<br>            SchemaDefinition=contract.to_avro()<br>        )<br>        return SchemaVersion(version=response[&#39;SchemaVersionId&#39;])<br>    <br>    def list_schemas(self) -&gt; List[Schema]:<br>        &quot;&quot;&quot;List all schemas.&quot;&quot;&quot;<br>        response = self.client.list_schemas()<br>        return [Schema.from_glue_response(s) for s in response[&#39;Schemas&#39;]]</pre><h3>Why AWS Glue?</h3><ol><li><strong>Managed service</strong>: No infrastructure to maintain</li><li><strong>Compatibility checking</strong>: Built-in schema evolution rules</li><li><strong>Industry standards</strong>: AVRO and Protobuf support</li><li><strong>Seamless integration</strong>: Works with AWS Glue ETL jobs</li><li><strong>Automatic versioning</strong>: Keeps track of all schema versions</li></ol><h3>Design Patterns &amp; Best Practices</h3><h3>1. Hexagonal Architecture (Ports &amp; Adapters)</h3><p>Benefits:</p><ul><li><strong>Testability</strong>: Business logic decoupled from frameworks</li><li><strong>Flexibility</strong>: Swap AWS Glue for another registry (Confluent Schema Registry, etc.)</li><li><strong>Clarity</strong>: Clear dependencies and responsibilities</li></ul><pre># Use cases depend on abstractions (ports), not concrete implementations<br>class RegisterSchemaUseCase:<br>    def __init__(self, schema_registry: SchemaRegistryPort):<br>        self.schema_registry = schema_registry<br>    <br>    def execute(self, contract: DataContract) -&gt; SchemaVersion:<br>        # Business logic here<br>        return self.schema_registry.register_schema(contract)</pre><h3>2. Dependency Injection</h3><p>The application wires all dependencies in app/main.py, making them explicit and testable:</p><pre>adapter = GlueSchemaRegistryAdapter()<br>use_case = RegisterSchemaUseCase(schema_registry=adapter)<br>router = create_router(register_schema_use_case=use_case)</pre><h3>3. Domain-Driven Design</h3><p>Business entities live in the domain/ layer:</p><pre># domain/models.py<br>class DataContract:<br>    contract_id: str<br>    name: str<br>    version: str<br>    columns: List[Column]<br>    metadata: Metadata<br>    <br>    def to_avro(self) -&gt; dict:<br>        &quot;&quot;&quot;Convert to AVRO schema.&quot;&quot;&quot;<br>        # Business logic for conversion<br>        pass</pre><h3>4. Separation of Concerns</h3><p>Each layer has a single responsibility:</p><ul><li><strong>Domain</strong>: “What is a data contract?” (business knowledge)</li><li><strong>Application</strong>: “How do I register a contract?” (orchestration)</li><li><strong>Adapters</strong>: “How do I talk to AWS Glue?” (technical details)</li></ul><h3>Running the System</h3><h3>Local Development</h3><pre># 1. Clone and setup<br>git clone git@github.com:vanikkars/schema-registry.git<br>cd schema-registry<br>cp .env.example .env<br>source .env</pre><pre># 2. Start services<br>docker-compose up</pre><pre># 3. Verify services<br>curl <a href="http://localhost:8000/health">http://localhost:8000/health</a><br>curl <a href="http://localhost:8001/health">http://localhost:8001/health</a></pre><pre># 4. Access interactive docs<br>open <a href="http://localhost:8000/docs">http://localhost:8000/docs</a></pre><h3>Registering a Schema</h3><pre># 1. Create contract file<br>cat &gt; contracts/current/product/product_v1.json &lt;&lt; &#39;EOF&#39;<br>{<br>  &quot;contract_id&quot;: &quot;product-v1&quot;,<br>  &quot;name&quot;: &quot;Product&quot;,<br>  &quot;version&quot;: &quot;1.0.0&quot;,<br>  &quot;columns&quot;: [<br>    {&quot;name&quot;: &quot;product_id&quot;, &quot;data_type&quot;: &quot;string&quot;, &quot;nullable&quot;: false},<br>    {&quot;name&quot;: &quot;name&quot;, &quot;data_type&quot;: &quot;string&quot;, &quot;nullable&quot;: false}<br>  ],<br>  &quot;metadata&quot;: {<br>    &quot;data_owner&quot;: &quot;Product Team&quot;,<br>    &quot;data_steward&quot;: &quot;Data Engineering&quot;<br>  }<br>}<br>EOF</pre><pre># 2. Register via API<br>curl -X POST &quot;http://localhost:8000/api/v1/schemas&quot; \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d @contracts/current/product/product_v1.json</pre><pre># 3. Create Iceberg table<br>curl -X POST &quot;http://localhost:8001/api/v1/tables&quot; \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{<br>    &quot;contract_id&quot;: &quot;product_v1&quot;,<br>    &quot;name&quot;: &quot;product&quot;,<br>    &quot;version&quot;: 1,<br>    &quot;columns&quot;: [<br>      {&quot;name&quot;: &quot;product_id&quot;, &quot;data_type&quot;: &quot;string&quot;},<br>      {&quot;name&quot;: &quot;name&quot;, &quot;data_type&quot;: &quot;string&quot;}<br>    ]<br>  }&#39;</pre><h3>Future Enhancements</h3><ol><li><strong>Schema Governance</strong>: Role-based access control for schema changes</li><li><strong>Impact Analysis</strong>: Show which datasets/pipelines depend on a schema</li><li><strong>Data Lineage</strong>: Track data flow through the organization</li><li><strong>Metric Validation</strong>: Assert data quality metrics alongside schemas</li><li><strong>Notification System</strong>: Slack/email alerts for breaking changes</li><li><strong>Schema Marketplace</strong>: Discover and browse all schemas in organization</li></ol><h3>Conclusion</h3><p>Data contracts are non-negotiable in modern data engineering. They establish trust between producers and consumers, prevent breaking changes, and create a foundation for reliable data systems. However, <strong>building a robust schema registry from scratch is a hard task</strong> with many nuances — schema versioning, compatibility checking, format conversion, and concurrent access all present subtle challenges that are easy to get wrong.</p><p>The key insight from this project is not to build your own schema registry, but to <strong>leverage production-ready solutions</strong>. AWS Glue Schema Registry and Confluent Schema Registry are battle-tested platforms that handle the complexities for you:</p><ul><li><strong>Compatibility checking</strong>: Implement complex rules (backward, forward, full compatibility) correctly</li><li><strong>Schema versioning</strong>: Manage versions reliably across distributed systems</li><li><strong>Format support</strong>: Handle AVRO, JSON, Protobuf seamlessly</li><li><strong>ACID guarantees</strong>: Ensure no data loss during schema transitions</li></ul><p>What <strong>you should build</strong> is the lightweight integration layer around these services — exactly what we’ve done here with FastAPI. This approach gives you:</p><ul><li><strong>Clean abstraction</strong>: A simple REST API that your team understands</li><li><strong>Native CI/CD integration</strong>: GitHub Actions automation that feels natural to your developers</li><li><strong>Flexibility</strong>: Swap AWS Glue for Confluent without changing your application code (thanks to hexagonal architecture)</li><li><strong>Reduced maintenance burden</strong>: Let AWS or Confluent handle the hard infrastructure problems</li></ul><p>By combining domain-driven design, clean architecture, and proven cloud solutions, we’ve created a system that puts data governance at the center of the data platform — without reinventing the wheel. Modern data infrastructure isn’t about building every component yourself; it’s about thoughtfully composing production-grade solutions with clean integration layers that make data reliability non-negotiable.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4f883d1b97f9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Observability for an Iceberg Lakehouse]]></title>
            <link>https://medium.com/@vanik_kars/observability-for-an-iceberg-lakehouse-b1bcbcceecef?source=rss-42fb2c934fcb------2</link>
            <guid isPermaLink="false">https://medium.com/p/b1bcbcceecef</guid>
            <category><![CDATA[lakehouse-architecture]]></category>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[apache-iceberg]]></category>
            <dc:creator><![CDATA[Vanik Karslian]]></dc:creator>
            <pubDate>Sun, 24 May 2026 09:57:45 GMT</pubDate>
            <atom:updated>2026-08-18T07:29:50.134Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eJHYc5Wg6WQaqBbX" /><figcaption>Photo by <a href="https://unsplash.com/@htn_films?utm_source=medium&amp;utm_medium=referral">Hubert Neufeld</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Full project in github: <a href="https://github.com/vanikkars/iceberg-monitoring">https://github.com/vanikkars/iceberg-monitoring</a></p><h3><strong>Intro</strong></h3><p>In previous <a href="https://medium.com/@vanik_kars/building-a-streaming-data-lakehouse-with-apache-flink-iceberg-and-nessie-c3c6a550ee21"><strong>article</strong></a> we have built a small streaming sourced <em>lakehouse</em> using Flink Kafka and Iceberg. This time I want to talk about something less glamorous but yet vital for production-ready <em>lakehouse</em>: <strong>observability</strong>. More specifically, we will focus solely on <strong>observability</strong> for Iceberg tables.</p><p>More often than not, I see even experienced software engineers getting lost in data-related topics. I have seen people repeatedly make basic mistakes when working with Iceberg:</p><ol><li><strong>Ignoring the small-files problem<br></strong>One of the <a href="https://www.dremio.com/blog/compaction-in-apache-iceberg-fine-tuning-your-iceberg-tables-data-files/">most</a> common mistakes is continuously generating tiny Parquet files without proper compaction. This leads to poor query performance, high metadata overhead, increased S3 API calls, and expensive compute costs.</li><li><strong>Not configuring snapshot expiration<br></strong>Teams often forget to expire old snapshots and orphan files. Over time, this causes <a href="https://www.linkedin.com/posts/dipankar-mazumdar_dataengineering-softwareengineering-share-7334013906190753792-YZXh/?utm_source=share&amp;utm_medium=member_desktop&amp;rcm=ACoAACKbYv0BV2JJCBNHNOcH_vqZ9GPvJzPNMv8">metadata bloat</a>, slower query planning, growing storage costs, and in extreme cases even unstable table operations.</li><li><strong>Using poor partitioning strategies<br></strong>Over-partitioning tables (for example, partitioning by high-cardinality columns such as user_id) creates too many small partitions and makes efficient file compaction almost impossible.</li></ol><p>I have seen production tables hundreds of GB in size with an average file size of only around 100 KB, the metadata.json bloated to 500MB in size and loaded into each query plan. The result was tremendously poor performance and skyrocketing infrastructure costs in live projects.</p><blockquote>You do not see it until you measure it.</blockquote><p>In order to mitigate such issues before they become critical, it is essential to have proper observability in place. Otherwise, you may only discover the problem once business teams start reaching out because a business-critical dashboard failed to load due to poor <em>lakehouse</em> performance. And trust me, that is not an experience you would appreciate.</p><p>So in this article I want to cover the <strong>key metrics you actually want to monitor on an Iceberg table.</strong></p><p>As usual, to try it out:</p><pre>git clone git@github.com:vanikkars/iceberg-monitoring.git<br>docker compose up --build</pre><h3>Recap of the Architecture</h3><p>For this little project I have prepared two Kafka topics that produce the source events:</p><ul><li>users</li><li>transactions</li></ul><p>A Flink SQL job consumes both and writes them into Iceberg tables every <strong>30 seconds</strong> (one Iceberg commit per Flink checkpoint). <br>The catalog is <strong>Apache Polaris</strong> speaking the Iceberg REST protocol, and the query engine is <strong>Trino</strong> with one coordinator and one worker — the less, is more!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kBn7YtDSozay3kbpmT9rZg.png" /><figcaption>High level architecture</figcaption></figure><h3>Iceberg Metadata Tables in Trino</h3><p>Iceberg exposes its own internal metadata statistics as <strong>regular SQL tables</strong>. In Trino you reach them by appending a $&lt;name&gt; suffix to the table:</p><pre>iceberg.demo.&quot;transactions$snapshots&quot;</pre><p>There are seven metadata tables that matter for day-to-day monitoring:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m1HMiBYVUi_VBo9HGmXfEw.png" /></figure><p>Fore a brief sanity check to verify if your data layout is “ok” you can check these two:</p><ul><li>$snapshots → commit cadence, records growth, file growth</li><li>$files → small-file problem, distribution of file sizes</li></ul><p>Let’s explore these metadata tables.</p><p><strong>Snapshots metadata table</strong></p><pre>SELECT<br>  *<br>FROM iceberg.demo.&quot;transactions$snapshots&quot;<br>ORDER BY committed_at DESC;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mZO6G7UVherYqzWewlE9Tw.png" /><figcaption>snapshots metadata table</figcaption></figure><p>As we can see, the last column is actually JSON and contains useful details about each snapshot, such as how many records were added or deleted, the total number of records, and more.</p><p>We can explore these details by simply extracting the relevant fields from the last column:</p><pre>SELECT<br>    snapshot_id,<br>    operation,<br>    CAST(element_at(summary, &#39;added-records&#39;)         AS bigint) AS added_records,<br>    CAST(element_at(summary, &#39;deleted-records&#39;)       AS bigint) AS deleted_records,<br>    CAST(element_at(summary, &#39;added-data-files&#39;)      AS bigint) AS added_data_files,<br>    CAST(element_at(summary, &#39;removed-data-files&#39;)    AS bigint) AS removed_data_files,<br>    CAST(element_at(summary, &#39;total-records&#39;)         AS bigint) AS total_records,<br>    CAST(element_at(summary, &#39;total-data-files&#39;)      AS bigint) AS total_data_files<br>FROM iceberg.demo.&quot;transactions$snapshots&quot;<br>ORDER BY committed_at DESC</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mw9RZxepOE3LYmiIyCAZyQ.png" /><figcaption>snapshots-related summary metrics</figcaption></figure><p>As you can see above, snapshots are committed roughly every 30 seconds (committed_at), and each commit operation is an append.</p><p>We can also observe how records are distributed across commits. For example, the snapshot committed at 2026-05-22 18:29:32.650000+00:00 added only 3,600 records. This can create small-file issues if it is not handled properly through Iceberg maintenance jobs.</p><p><strong>Files metadata table</strong></p><pre>SELECT<br>    file_format,<br>    count(*)                                            AS file_count,<br>    sum(record_count)                                   AS total_records,<br>    round(avg(record_count), 0)                         AS avg_records_per_file,<br>    min(record_count)                                   AS min_records,<br>    max(record_count)                                   AS max_records,<br>    sum(file_size_in_bytes)                             AS total_size_bytes,<br>    round(avg(file_size_in_bytes) / 1024.0 / 1024, 2)  AS avg_size_mb<br>FROM iceberg.demo.&quot;transactions$files&quot;<br>GROUP BY file_format</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/1*oDeXbDC8tnteAbaUxSH1QA.png" /><figcaption>Files table</figcaption></figure><p>This query provides a high-level overview of the average file size and the number of records stored in each Parquet file backing the transactions table.</p><p>As the output reveals, the average file size is only 1.85 MB, which is a strong indicator of a <strong>small-files problem</strong>.</p><p>These were just two examples demonstrating the power of Iceberg metadata tables. More details, along with the full list of exploration queries, are available in Jupyter notebook: <a href="https://github.com/vanikkars/iceberg-monitoring/blob/main/docker/jupyter/notebooks/trino_iceberg_metadata.ipynb">https://github.com/vanikkars/iceberg-monitoring/blob/main/docker/jupyter/notebooks/trino_iceberg_metadata.ipynb</a>, so feel free to check them out.</p><p>So far, we have explored these metadata queries in an ad-hoc manner. However, in production environments, you would typically want <strong>observability dashboards</strong> that expose these metrics through a clean UI</p><p>— so welcome to Grafana!</p><h3>What Is Grafana?</h3><p>For anyone who has not used it: Grafana is an open-source dashboarding tool. It supports wide range of sources including Prometheus, InfluxDB, SQL databases, or in our case Trino.<br>Grafana renders the results as panels (time series, tables, gauges, stat tiles). It also has a templating system so you can parameterise dashboards with dropdown variables.</p><h3>Dashboards</h3><p>In our example <strong>Grafana</strong> dashboard we have high level metrics on top showing the aggregated totals per table:</p><ul><li>number of records</li><li>number of commits</li><li>avg file size</li><li>time from the last commit</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HI4Ol_YJfAhDoP4FHHOVjQ.png" /><figcaption>aggregated metrics per table</figcaption></figure><p>The <em>Commit Activity</em> section provides a historical overview of commits made to each table over time. In particular, it shows how the total number of rows grows over time and how frequently data is committed to the corresponding Iceberg table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bBLax_OGWoEwoq0cJsggXw.png" /><figcaption>time retrospective view</figcaption></figure><p>Particularly interesting is the <em>Files</em> section, which shows aggregated file-level metrics (such as average file size), as well as a timeline illustrating how file sizes evolve over time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vkq2-MHH7YqGSkPKTneFMw.png" /><figcaption>File level metrics</figcaption></figure><p>Funny enough while I was finishing my writing the file size in transactions has decreased from 1.85 MB to 604 KB, stating even stronger the problem of small files :D</p><h3><strong>Summary</strong></h3><p>In this article, I tried to show which Iceberg table metrics are worth monitoring in your <em>lakehouse</em> in order to avoid future headaches related to small-file issues or snapshot bloat.</p><p>In the next article, I will dive deeper into the impact small files can have on your <em>lakehouse</em> and explore different strategies for solving this problem.</p><p>For more details and additional examples follow me on <a href="https://github.com/vanikkars">GitHub</a> and <a href="https://www.linkedin.com/in/vanik-karslian/">Linkedin</a>.</p><p>Thank you for reading all the way to the end!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b1bcbcceecef" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Streaming Data Lakehouse with Apache Flink, Iceberg, and Nessie]]></title>
            <link>https://medium.com/@vanik_kars/building-a-streaming-data-lakehouse-with-apache-flink-iceberg-and-nessie-c3c6a550ee21?source=rss-42fb2c934fcb------2</link>
            <guid isPermaLink="false">https://medium.com/p/c3c6a550ee21</guid>
            <category><![CDATA[modern-data-architecture]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[apache-iceberg]]></category>
            <category><![CDATA[data-platforms]]></category>
            <dc:creator><![CDATA[Vanik Karslian]]></dc:creator>
            <pubDate>Wed, 29 Apr 2026 17:36:01 GMT</pubDate>
            <atom:updated>2026-05-02T19:46:39.641Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2FTalEvqLedCSKt9" /><figcaption>Photo by <a href="https://unsplash.com/@lucabravo?utm_source=medium&amp;utm_medium=referral">Luca Bravo</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>full project in github: <a href="https://github.com/vanikkars/flink-ingestion">https://github.com/vanikkars/flink-ingestion</a></p><h3><strong>Why I Built This</strong></h3><p>Current “modern data stack” can not be imagined without <strong>lakehouse</strong>. <br>The <strong>lakehouse</strong> architecture aims to bridge the gap between traditional <strong>data lakes</strong> and <strong>data warehouses</strong>. It combines the strengths of both approaches by providing ACID guarantees, data versioning, and time travel capabilities, while still maintaining high scalability and cost efficiency. At the same time, it avoids reliance on proprietary services, helping reduce vendor lock-in.</p><p>In this project, I aim to demonstrate how a <strong>lakehouse</strong> architecture can support time-sensitive business use cases, which are especially common in the FinTech and banking industries, through <strong>near real-time</strong> data pipelines.</p><p>At the same time, I wanted to keep the project lightweight and easy to run locally, so anyone can experiment with it on their laptop. So I could share it with my team and say — just clone this repo and run <em>docker compose up</em>.</p><p>The full stack: <strong>Apache Kafka → Apache Flink → Apache Iceberg → Nessie → DuckDB → Evidence BI/Jupyter Lab</strong>.</p><h3><strong>The architecture</strong></h3><p><strong>High level<br></strong>The <strong>high level</strong> architecture of the project is shown on the diagram below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UpZR3AdgUWbjhhvXVVCh5Q.png" /><figcaption>High level architecture</figcaption></figure><p>In the nutshell, a <strong>python producer</strong> generates synthetic users and transactions at a configurable rate (default: 2 events/sec) and publishes the messages into <strong>Kafka topics</strong>. <br><strong>Flink</strong> reads from Kafka via SQL, applies watermarks, writes into <strong>Iceberg tables </strong>and registers them in <strong>Nessie Catalog</strong>. <br><strong>DuckDB</strong> is linked via <a href="https://duckdb.org/docs/current/core_extensions/iceberg/iceberg_rest_catalogs"><strong>Iceberg extension</strong></a> to Nessie catalog and offers query capabilities for the downstream consumers — the <a href="https://evidence.dev/"><strong>Evidence BI tool</strong></a> and Jupyter lab.</p><p><strong>Data Ingestion with Apache Flink</strong></p><p>For many newcomers, <strong>Apache Flink</strong> comes with a steep learning curve. This is especially true for <strong>data practitioners</strong> who primarily work with <strong>Python</strong> and <strong>SQL</strong> for whom the Java and Scala -based implementations can introduce unnecessary friction.</p><p>Therefore, in this project I aim to showcase the SQL capabilities of Apache Flink, which significantly lower the barrier for <strong>data</strong> and <strong>analytics</strong> engineers to define ingestion pipelines.</p><p>In essence, you can define a data pipeline <strong>Apache Kafka</strong> to <strong>Apache Iceberg tables</strong> — using just a few SQL statements:</p><ol><li>as a first step the <strong>kafka</strong> source has to be defined:</li></ol><pre>CREATE TEMPORARY TABLE kafka_users (<br>    user_id     STRING,<br>    name        STRING,<br>    email       STRING,<br>    country     STRING,<br>    created_at  TIMESTAMP(3),<br>    WATERMARK FOR created_at AS created_at - INTERVAL &#39;5&#39; SECOND<br>) WITH (<br>    &#39;connector&#39;                      = &#39;kafka&#39;,<br>    &#39;topic&#39;                          = &#39;users&#39;,<br>    &#39;properties.bootstrap.servers&#39;   = &#39;kafka:29092&#39;,<br>    &#39;properties.group.id&#39;            = &#39;flink-iceberg-users&#39;,<br>    &#39;scan.startup.mode&#39;              = &#39;earliest-offset&#39;,<br>    &#39;format&#39;                         = &#39;json&#39;,<br>    &#39;json.timestamp-format.standard&#39; = &#39;ISO-8601&#39;<br>);</pre><p>2. define the target tagble in <strong>Iceberg</strong>:</p><pre>CREATE TABLE IF NOT EXISTS iceberg_catalog.demo.users (<br>    user_id     STRING,<br>    name        STRING,<br>    email       STRING,<br>    country     STRING,<br>    created_at  TIMESTAMP(3)<br>) WITH (<br>    &#39;write.format.default&#39; = &#39;parquet&#39;,<br>    &#39;write.upsert.enabled&#39; = &#39;false&#39;<br>);</pre><p>3. launch the ingestion from <strong>kafka</strong> to I<strong>ceberg</strong>:</p><pre>INSERT INTO iceberg_catalog.demo.users<br>SELECT user_id, name, email, country, created_at<br>FROM kafka_users;</pre><p>With just three SQL statements, your data is synced into Iceberg — what could be simpler? No complex Java, Scala, or even Python code is required.</p><p>More details: <a href="https://github.com/vanikkars/flink-ingestion/blob/main/docker/flink/jobs/sql/kafka_to_iceberg.sql"><strong>docker/flink/jobs/sql/kafka_to_iceberg.sql</strong></a></p><p><strong>Data Catalog</strong></p><p>A data catalog is a core component of an <strong>Iceberg Lakehouse</strong>. In this project, I focus on <a href="https://projectnessie.org/">Project Nessie</a>, which provides multiple interfaces (including REST API) and powerful features such as a Git-like catalog with branches, tags, and merges for your data.</p><p>Its REST API makes the setup catalog-agnostic — tools like Apache Flink, DuckDB, and Apache Spark can all connect to the same catalog natively (at the time of writing <a href="https://duckdb.org/docs/current/core_extensions/iceberg/iceberg_rest_catalogs"><strong>DuckDB iceberg extension</strong></a> does only support REST API).</p><p><strong>Query Engine</strong></p><p>To enable query capabilities for BI dashboards and support ad-hoc data exploration, a query engine is required. For this project, I chose <a href="https://duckdb.org/"><strong>DuckDB</strong></a><strong> </strong>— a modern <strong>“Swiss Army knife”</strong> for data engineering. It provides support for <strong>Apache Iceberg</strong> tables via its <a href="https://duckdb.org/docs/current/core_extensions/iceberg/overview"><strong>Iceberg extension</strong></a> and integrates natively with REST-based data catalogs.</p><p>Here is how the configuration of duckDB for <strong>Jupyter notebook</strong> looks like</p><ol><li>first we need to import the duckDB python library and install the required extensions</li></ol><pre>import duckdb<br><br>con = duckdb.connect()<br><br>con.execute(&quot;INSTALL httpfs&quot;)<br>con.execute(&quot;INSTALL iceberg&quot;)<br>con.execute(&quot;LOAD httpfs&quot;)<br>con.execute(&quot;LOAD iceberg&quot;)<br></pre><p>2. then define the secrets to authorise duckDB in s3 (minio):</p><pre># S3 credentials - DuckDB uses these to sign its own S3 requests directly<br>con.execute(&quot;&quot;&quot;<br>CREATE OR REPLACE SECRET minio (<br>    TYPE        s3,<br>    KEY_ID      &#39;minioadmin&#39;,<br>    SECRET      &#39;minioadmin&#39;,<br>    ENDPOINT    &#39;minio:9000&#39;,<br>    URL_STYLE   &#39;path&#39;,<br>    USE_SSL     false,<br>    REGION      &#39;us-east-1&#39;<br>)<br>&quot;&quot;&quot;)</pre><p>3. then provide the configurations for Nessie Catalog:</p><pre># ACCESS_DELEGATION_MODE &#39;none&#39; tells DuckDB to skip Nessie&#39;s s3sign endpoint<br># and sign S3 requests itself using the secret above<br>con.execute(&quot;&quot;&quot;<br>ATTACH &#39;demo_lh&#39; AS nessie (<br>    TYPE                   iceberg,<br>    ENDPOINT               &#39;http://nessie:19120/iceberg&#39;,<br>    AUTHORIZATION_TYPE     &#39;none&#39;,<br>    ACCESS_DELEGATION_MODE &#39;none&#39;<br>)<br>&quot;&quot;&quot;)<br><br>con.execute(&quot;SHOW ALL TABLES&quot;).df()</pre><p>and voila — duckDB is crunching the iceberg tables!</p><p>Full setup: <a href="https://github.com/vanikkars/flink-ingestion/blob/main/docker/jupyter/notebooks/query_iceberg.ipynb"><strong>docker/jupyter/notebooks/query_iceberg.ipynb</strong></a></p><p><strong>BI dashboards via Evidence BI</strong></p><p><a href="https://docs.evidence.studio/"><strong>Evidence</strong></a> is a modern, lightweight Business Intelligence tool that follows a “BI as code” approach. The whole BI is just files in a directory that you can version control.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/427/1*IG6tGgXwQOgubmsKIMrBKg.png" /></figure><p>Dashboards are defined using Markdown files, while SQL queries are written in .sql files and then referenced within the dashboard .md pages.</p><p>Here is an example of the index page that contains high level KPIs and links to detailed transaction and users dashboards:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/1*bJ7npEUm3HKwRPgODlWN_g.png" /><figcaption>The BI main page</figcaption></figure><p>The image below shows the transactions dashboard:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UC045YK5K3lAT2CtYv8kXw.png" /><figcaption>The transactions dashboard</figcaption></figure><h3><strong>Further improvements</strong></h3><ol><li><strong>Partitioning Strategy on Iceberg Tables </strong>— define explicit partition specs on the Iceberg tables to reduce scan footprints, improve query performance, and enable efficient partition pruning in downstream analytics.</li><li><strong>Regular Optimization Jobs</strong> (Compaction, Sorting, etc.) — schedule periodic maintenance jobs using Iceberg’s rewrite_data_files and rewrite_manifests procedures to compact small files generated by streaming ingestion, apply Z-ordering or sorting on frequently filtered columns, and expire old snapshots to control storage growth.</li><li><strong>Observability over Iceberg Metrics</strong> — instrument the pipeline to expose Iceberg table-level metrics such as:<br>- snapshot counts<br>- file counts<br>- total records<br>- manifest size<br>- compaction lag</li></ol><h3><strong>Running It Yourself</strong></h3><pre>git clone git@github.com:vanikkars/flink-ingestion.git<br><br>docker compose up --build</pre><p>Wait about 60 seconds for Flink to submit the job and the first checkpoint to commit.</p><ul><li>Evidence Dashboard <a href="http://localhost:3000/">http://localhost:3000</a></li><li>Flink UI <a href="http://localhost:8081/">http://localhost:8081</a></li><li>JupyterLab <a href="http://localhost:8888/">http://localhost:8888</a></li><li>MinIO Console <a href="http://localhost:9001/">http://localhost:9001</a> (minioadmin / minioadmin)</li><li>Nessie API <a href="http://localhost:19120/iceberg/v1/config">http://localhost:19120/iceberg/v1/config</a></li></ul><h3><strong>Final Thoughts</strong></h3><p>The open lakehouse stack has matured significantly. Technologies like Apache Flink SQL, Apache Iceberg, and Project Nessie together provide a solid foundation for getting started quickly.<br>However, moving such a setup into production requires considerably more effort which is out of the scope of the current project. <br>Beyond the initial deployment, teams need to invest in reliable job orchestration, robust maintenance workflows, and comprehensive observability. This includes monitoring iceberg tables state, tracking maintenance and compaction jobs, and understanding access patterns to ensure performance and stability over time.</p><p>Hopefully this saves someone that time.</p><p>The full project is on <a href="https://github.com/vanikkars/flink-ingestion/"><strong>GitHub</strong></a>. PRs and issues welcome.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c3c6a550ee21" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>