<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Shreya Arora | Data Engineering on Medium]]></title>
        <description><![CDATA[Stories by Shreya Arora | Data Engineering on Medium]]></description>
        <link>https://medium.com/@shreya16april?source=rss-a4910360d908------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*NcsczWJ6MlF-ohPQ.jpg</url>
            <title>Stories by Shreya Arora | Data Engineering on Medium</title>
            <link>https://medium.com/@shreya16april?source=rss-a4910360d908------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 22:07:33 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shreya16april/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Rethinking Medallion Architecture for Commercial Banking: How We Built SWIFT Flow Intelligence in…]]></title>
            <link>https://medium.com/@shreya16april/rethinking-medallion-architecture-for-commercial-banking-how-we-built-swift-flow-intelligence-in-830223981396?source=rss-a4910360d908------2</link>
            <guid isPermaLink="false">https://medium.com/p/830223981396</guid>
            <category><![CDATA[medallion-architecture]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-curation]]></category>
            <category><![CDATA[data-modelling]]></category>
            <category><![CDATA[data-profiling]]></category>
            <dc:creator><![CDATA[Shreya Arora | Data Engineering]]></dc:creator>
            <pubDate>Thu, 19 Nov 2026 17:46:36 GMT</pubDate>
            <atom:updated>2026-11-19T17:46:36.552Z</atom:updated>
            <content:encoded><![CDATA[<h3>Rethinking Medallion Architecture for Commercial Banking: How We Built SWIFT Flow Intelligence in Dataiku</h3><p>The Medallion architecture — Bronze, Silver, and Gold — quickly became the default blueprint for data lakehouse design, and for good reason. It brought order to unstructured data chaos, enforced linear data progression, and provided reliable lineage for enterprise reporting.</p><p>However, as modern banking data requirements evolve from static end-of-month reporting to real-time counterparty intelligence and proactive client retention, the classical Medallion model faces real operational friction.</p><p>Here is an architectural look at how we implemented a Medallion-inspired pipeline inside Dataiku to analyze commercial client behavior, map external SWIFT fund flows, and trigger hyper-targeted treasury offers — alongside the practical lessons learned where the architecture begins to strain.</p><h3>The Core Challenge: Tracking Money Leakage and Cross-Bank Flows</h3><p>In commercial banking, understanding money movement is everything. When a corporate client routinely routes high-volume SWIFT messages or wire transfers to external financial institutions, it represents two things:</p><ol><li><strong>Transaction Leakage:</strong> Capital moving out of your balance sheet to competitor banks.</li><li><strong>Opportunity:</strong> A signal that the client has treasury, foreign exchange, or liquidity needs that your institution is currently not capturing.</li></ol><p>To turn raw settlement logs into actionable commercial strategy, we needed a scalable data pipeline capable of processing complex transactional data, resolving messy enterprise client entities, and delivering model-ready features for offer generation.</p><h3>Architectural Breakdown: Building the Flow in Dataiku</h3><p>Using Dataiku’s orchestration and visual recipe engine paired with pushdown SQL compute, we organized our data environment into distinct processing tiers aligned with the Medallion framework.</p><pre>+-----------------------------------------------------------------------------------+<br>| BRONZE LAYER (Raw Audit Zone)                                                     |<br>| - Ingestion of raw Core Banking ledger tables &amp; SWIFT MT103 / ISO 20022 messages  |<br>| - Schema enforcement at landing; payload preserved for full replayability         |<br>+-----------------------------------------------------------------------------------+<br>                                          │<br>                                          ▼<br>+-----------------------------------------------------------------------------------+<br>| SILVER LAYER (Entity Resolution &amp; Cleansed Messaging)                             |<br>| - Standardized ISO messaging (sender/receiver BIC parsing, IBAN normalization)    |<br>| - Deduplicated commercial entity identity graph &amp; client relationship maps        |<br>+-----------------------------------------------------------------------------------+<br>                                          │<br>                                          ▼<br>+-----------------------------------------------------------------------------------+<br>| GOLD LAYER (Commercial Intelligence &amp; Offer Engine)                               |<br>| - Outward Flow Aggregations (Volume, Frequency, Recipient Bank BICs)              |<br>| - Commercial Retention Signals &amp; Personalized Offer Generation Models             |<br>+-----------------------------------------------------------------------------------+</pre><h3>1. The Bronze Layer: Auditable Ingestion</h3><p>The Bronze layer acts as the immutable landing strip for high-volume, heterogeneous transaction payloads.</p><ul><li><strong>Data Sources:</strong> Raw core banking ledger feeds, internal transfer logs, and raw SWIFT messaging files (MT103 / ISO 20022 XML formats).</li><li><strong>Implementation:</strong> Ingested into Dataiku without business transformations, appending technical audit metadata (ingestion_timestamp, source_system_id, batch_id).</li><li><strong>Why It Matters:</strong> In financial services, payload preservation is essential for regulatory auditing and back-testing historical data models.</li></ul><h3>2. The Silver Layer: Normalization &amp; Commercial Entity entity. A single corporate client might hold twenty accounts across sub-entities using slightly different legal names.</h3><ul><li><strong>SWIFT Parsing:</strong> Extracting structured fields from unstructured/semi-structured wire notes — isolating Bank Identifier Codes (BICs), routing details, IBANs, and remittance codes.</li><li><strong>Entity Mapping:</strong> Linking fragmented sender accounts to a unified Commercial Client Master using Dataiku’s fuzzy matching, join recipes, and custom Python routines.</li><li><strong>Result:</strong> A clean, deduplicated transactional layer where every outgoing wire or SWIFT transfer is accurately mapped to a verified commercial account holder.</li></ul><h3>3. The Gold Layer: Counterparty Analytics &amp; Retention Triggering</h3><p>The Gold layer shifts the focus from data hygiene to business monetization. Here, transactional networks are translated into client propensity models and actionable relationship manager insights.</p><ul><li><strong>Behavioral Profiling:</strong> Aggregating metrics across custom rolling windows (e.g., 30/60/90-day outbound volumes to specific competitor BICs, transfer frequency spikes, and average payment sizes).</li><li><strong>Targeted Commercial Incentives:</strong> Automated logic identifies clients leaking significant transaction volume. When a threshold is met, the pipeline generates personalized commercial offers — such as preferential FX rates, discounted treasury management packages, or tailored fee structures — designed to pull deposits back onto the balance sheet.</li></ul><h3>Where Medallion Starts to Crack in Financial Analytics</h3><p>While this architecture delivered significant business value, applying strict Medallion principles to complex financial graphs highlighted a few architectural limitations:</p><h3>1. Storage Overhead from Rigid Layering</h3><p>Persisting massive transaction logs across three separate physical datasets (Bronze, Silver, Gold) creates substantial data duplication. Querying large-scale counterparty matrices dynamically often makes static Gold tables stale the moment they are written.</p><h3>2. Latency vs. Batch Processing</h3><p>Traditional Medallion pipelines rely on scheduled batch processing. However, commercial retention signals lose impact over time. Waiting for an end-of-day Gold batch to identify a major outgoing treasury transfer delays the window for automated intervention.</p><h3>3. Schema Rigidity with Modern AI/LLM Workloads</h3><p>Standard relational Gold tables work well for BI dashboards, but modern client intelligence increasingly requires unstructured data — such as payment memo context or sentiment analysis from client interaction logs. Standardizing this varied data into classic schema-bound Gold tables adds maintenance complexity.</p><h3>Looking Ahead: Evolving Toward the Medallion Mesh</h3><p>To address these pain points, data architectures in banking are evolving beyond strict linear pipelines toward a <strong>Medallion Mesh</strong>:</p><ul><li><strong>Semantic Virtualization:</strong> Leveraging semantic layers to build dynamic, virtual Gold views directly on top of Silver data, reducing storage costs and pipeline latency.</li><li><strong>Real-Time Feature Stores:</strong> Streaming Silver events into online feature stores to trigger commercial retention offers within operational workflows rather than overnight batch jobs.</li><li><strong>Domain Ownership:</strong> Decentralizing data operations so commercial treasury teams manage their own domain models end-to-end inside platforms like Dataiku.</li></ul><h3>Conclusion</h3><p>The Medallion pattern provided a strong framework for moving from raw, messy transaction feeds to high-value commercial intelligence. However, as business needs shift toward real-time engagement and dynamic graph analysis, data teams must treat Medallion as a flexible guide rather than a rigid set of rules.</p><p><em>What patterns are you using to manage entity resolution and real-time transaction analytics in your platform? Let’s connect in the comments.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=830223981396" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Rebuilding a Multi-Job Data Pipeline as a Star Schema: ~8x Faster, Zero Drift]]></title>
            <link>https://medium.com/@shreya16april/rebuilding-a-multi-job-data-pipeline-as-a-star-schema-8x-faster-zero-drift-588645bc350f?source=rss-a4910360d908------2</link>
            <guid isPermaLink="false">https://medium.com/p/588645bc350f</guid>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[data-modeling]]></category>
            <category><![CDATA[data-architecture]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[analytics-engineering]]></category>
            <dc:creator><![CDATA[Shreya Arora | Data Engineering]]></dc:creator>
            <pubDate>Thu, 19 Nov 2026 16:49:36 GMT</pubDate>
            <atom:updated>2026-11-19T16:49:36.980Z</atom:updated>
            <content:encoded><![CDATA[<p><em>How I replaced a chain of jobs with a single dimensional-model job, and what I learned designing it end to end.</em></p><h3>The problem</h3><p>A common pattern in analytics engineering: a pipeline grows one job at a time. Mine tracked a user conversion funnel from raw analytics events, and it had become a chain of four jobs:</p><ul><li>Two jobs ran in parallel, each applying a different set of business rules to the same raw data.</li><li>A third job merged their outputs.</li><li>A fourth job aggregated everything to session level.</li></ul><p>It produced correct results, but it was wasteful. The raw data was read twice. Every run wrote several large, wide, event-level intermediate tables whose only purpose was to hand data from one job to the next. And four jobs meant four things to schedule, monitor, and debug.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s1DsDoivfATbzjM1X-nU5g.png" /><figcaption>Replacing a chain of jobs with a single star-schema job.</figcaption></figure><h3>The design</h3><p>I replaced the chain with <strong>one job built around a dimensional model</strong>. These were the key design decisions.</p><h3>1. Read the source once</h3><p>Only the small set of columns the funnel actually needs is pulled, a single time. Narrow reads are one of the cheapest wins in any pipeline.</p><h3>2. Apply business rules in memory</h3><p>Both rule sets run at event level inside the same job. Nothing is written to disk between steps, which removes the intermediate tables entirely.</p><h3>3. Roll up to session level in memory</h3><p>Events collapse to one row per session and date before anything is persisted. No event-level table is ever stored, only compact session-level data.</p><h3>4. Fan out into a star schema</h3><p>From the rolled-up data, the job writes:</p><ul><li><strong>Conformed dimensions</strong> (such as date, channel, and customer segment) with <strong>integer surrogate keys</strong></li><li>A <strong>daily fact table</strong> partitioned by event date, and a <strong>monthly fact table</strong> partitioned by month</li><li><strong>Physical reporting tables</strong> (facts joined to dimensions) for fast, simple downstream queries</li><li><strong>Views</strong> over the same joins for flexible access</li></ul><h3>5. Make it idempotent by design</h3><ul><li>Fact tables <strong>overwrite only the partitions being processed</strong>, so reruns and backfills are safe.</li><li>Dimensions are <strong>append-only</strong>, so surrogate keys never change and historical joins never break.</li></ul><h3>6. One codebase for daily runs and backfills</h3><p>The same code handles the daily run and a monthly backfill mode, orchestrated as a single task. The code is split by responsibility (source and rules, dimensions, facts, job flow), and paths and table names live in configuration rather than in code.</p><h3>Proving it didn’t break anything</h3><p>A faster pipeline is worthless if the numbers drift. I ran the old and new pipelines side by side on real production data and compared them row by row across every funnel metric and every dimension attribute. The result was <strong>zero mismatches</strong>.</p><p>This step is what turns a rewrite from a risky claim into something stakeholders can trust.</p><h3>The results</h3><p>In my testing, the new design delivered:</p><ul><li><strong>Roughly 8x faster</strong> end-to-end runtime</li><li><strong>One job instead of four</strong> to schedule and monitor</li><li><strong>One read of the raw source instead of two</strong></li><li><strong>No event-level intermediate tables</strong>, so a much smaller storage footprint</li><li><strong>Identical output</strong>, verified row for row</li></ul><h3>Takeaways</h3><ul><li><strong>Don’t persist what you can compute in memory.</strong> Intermediate tables are often a habit, not a requirement.</li><li><strong>Model for the consumer.</strong> Facts, dimensions, and reporting tables serve many users better than one wide denormalized table.</li><li><strong>Design for reruns first.</strong> Partition overwrites and append-only dimensions make backfills boring, which is exactly what you want.</li><li><strong>Validate before you celebrate.</strong> A row-for-row comparison on production-scale data is what earns trust for a rewrite.</li><li><strong>Fewer moving parts is a feature.</strong> Collapsing four jobs into one improved speed and also simplified operations and debugging.</li></ul><p><em>If you’re untangling a chain of jobs into a cleaner model, I’d love to hear how you approached it.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=588645bc350f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>