<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Stanislav Tvarůžek on Medium]]></title>
        <description><![CDATA[Stories by Stanislav Tvarůžek on Medium]]></description>
        <link>https://medium.com/@stanislav.tvaruzek?source=rss-195a54f44981------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*rRWKv3wwiMN9XAsUES5qIQ.jpeg</url>
            <title>Stories by Stanislav Tvarůžek on Medium</title>
            <link>https://medium.com/@stanislav.tvaruzek?source=rss-195a54f44981------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 16:09:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@stanislav.tvaruzek/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Processing Oracle HCM Extracts: From a Single Integration to a Global Rollout]]></title>
            <link>https://medium.com/@stanislav.tvaruzek/processing-oracle-hcm-extracts-from-a-single-integration-to-a-global-rollout-9199317c080b?source=rss-195a54f44981------2</link>
            <guid isPermaLink="false">https://medium.com/p/9199317c080b</guid>
            <category><![CDATA[payroll]]></category>
            <category><![CDATA[oracle-hcm]]></category>
            <category><![CDATA[message-queue]]></category>
            <category><![CDATA[integration]]></category>
            <category><![CDATA[webmethod]]></category>
            <dc:creator><![CDATA[Stanislav Tvarůžek]]></dc:creator>
            <pubDate>Fri, 18 Sep 2026 17:43:24 GMT</pubDate>
            <atom:updated>2026-09-18T17:43:24.543Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GpEePeVf78CpqZNjTWqsZw.png" /></figure><blockquote><em>A Practical Blueprint for Scalable, Reliable Payroll Data Processing</em></blockquote><h3>Introduction</h3><p>At first glance, connecting a global human capital management platform to payroll providers seems simple: extract, transform, and load employee data.</p><p>In reality, HCM extracts are large, HR data is complex and deeply layered, every country has its own requirements, and payroll providers need different formats and protocols.</p><p>This article explains the high-level architecture I created to deliver Oracle HCM extract data through webMethods middleware to multiple payroll and non-payroll systems.</p><p>You can reuse the principles and patterns here on other integration platforms, and the lessons apply to any large-scale payroll integration project.</p><h3>Who This Article Is For</h3><p>This article is for integration architects, integration developers, payroll technology leads, Oracle HCM consultants, and anyone dealing with large files or high-volume batch integrations.</p><p>If you are facing challenges with scaling HCM extracts, want to improve data integration reliability, or are looking for proven architectural patterns, you will find useful insights here.</p><p>This article assumes you are familiar with Oracle HCM, payroll systems, and enterprise integration concepts. It is not an implementation guide.</p><p>Instead, it provides a high-level overview of the architecture, key design decisions, trade-offs, and lessons learned.</p><p>The integration was built on webMethods, but the principles and patterns apply to other integration platforms.</p><p>The focus is on designing a scalable, reliable, and reusable architecture to process large HCM extracts and deliver data to multiple payroll and non-payroll systems.</p><p>The same architectural pattern applies beyond payroll and Oracle HCM whenever large source files must be processed efficiently and reliably.</p><p>Domain-specific terms are defined in the glossary at the end of the article.</p><h3>The Integration Landscape at a Glance</h3><p>This project evolved from a single HCM-to-payroll interface into a global one-to-many integration platform.</p><p>Oracle HCM provided large employee extracts to the middleware, which distributed relevant data to various payroll and non-payroll systems, both internal and external.</p><p>Each destination could have distinct populations, business rules, message formats, delivery protocols, and schedules.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7RuETHvjsQGZtfB2iX4SLA.png" /></figure><h3>How the Architecture Took Shape</h3><p>I did not start with a blank sheet of paper. I began by reviewing existing HCM integrations and noticed three major issues to address.</p><p>First, each payroll integration implemented its own inbound flow, duplicating the same extract lifecycle.</p><p>Secondly, each integration processed the entire HCM extract in memory.</p><p>Lastly, the integrations ran synchronously end to end.</p><p>Let’s address these issues one by one and see how the architecture evolved.</p><h3>The Inbound Flow</h3><p>The inbound flow handled data exchange between Oracle HCM and the middleware.</p><h3>Orchestrating Oracle HCM Extracts</h3><p>Generating an Oracle HCM extract is asynchronous. A request returns an extract-run identifier, not the finished file.</p><p>Persistence is essential because HCM extract generation can take minutes or hours, network calls can fail, and servers can restart. Processing must resume from a durable state.</p><p>The integration platform therefore manages each run as a persistent state machine:</p><ol><li>A scheduler initiates the extract run.</li><li>The platform submits the extract request to Oracle HCM.</li><li>Oracle returns the extract-run ID.</li><li>The platform stores the ID and current state.</li><li>A scheduled process polls Oracle for completion.</li><li>The platform downloads the extract.</li><li>The platform decrypts and processes the extract.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pxp42ySDtKflPrBQqu58sg.png" /></figure><h3>Runtime Integration Configuration</h3><p>The very first thing I did was to streamline the inbound flow and make it reusable for all payroll integrations.</p><p>To make it work, the inbound flow had to be configurable at runtime.</p><p>Instead of hard-coding extract parameters in each integration, the inbound flow reads an integration configuration file, passes the parameters to Oracle HCM, and then handles the extract lifecycle the same way for all integrations.</p><p>This allowed the same inbound flow to handle multiple HCM extracts with different extract definitions, populations, and effective-date behaviors.</p><p>In addition, the integration XML files could hold integration-specific settings, such as timeouts, retry counts, and feature switches.</p><p>Every integration had an XML configuration. The integration ID was passed to the scheduler that initiated a run.</p><p>This ID became the entry point for runtime behavior.</p><p>The following diagram illustrates the inbound flow lifecycle, including configuration lookup.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*beTnVaV0kJ92JNrSoPw8eg.png" /></figure><p>The scheduler initiated the payroll run and passed the integration ID to the inbound flow. The inbound flow looked up the integration configuration, submitted the extract request to Oracle HCM, polled for completion, downloaded the extract, and decrypted it.</p><p>Let’s define a sample integration configuration for Integration A:</p><pre>&lt;integration&gt;<br>    &lt;id&gt;A&lt;/id&gt;<br><br>    &lt;!-- hcm extract parameters --&gt;<br>    &lt;hcmExtract&gt;<br>      &lt;id&gt;HCM_EMPLOYEE_EXTRACT&lt;/id&gt;<br>      &lt;country&gt;XY&lt;/country&gt;<br>      &lt;effectiveDate&gt;XYZ&lt;/effectiveDate&gt;<br>      &lt;!-- other parameters --&gt;<br>    &lt;/hcmExtract&gt;<br>  &lt;/integration&gt;</pre><p>That’s all we need now in the integration configuration. We can add other parameters like business unit or legal entity to further filter the population.</p><p>The important result was a parameterized inbound flow that could accommodate different HCM extract parameters and be fine-tuned without code changes.</p><p>Let me recap. We submitted an HCM extract, retrieved its content, and decrypted it. The extract is sitting on a shared file system.</p><p>Now let’s step away from the middleware and move back to HCM. We need to understand the extraction modes and the constraints that shaped the architecture.</p><h3>HCM Extraction Strategy: CHANGES_ONLY vs. BLOCK_OLD</h3><p>Highly competent HCM domain experts designed the HCM extract definitions and advised on the best extraction mode for our integrations.</p><p>The existing integrations I reviewed used CHANGES_ONLY mode. In that mode, Oracle HCM determines which attributes changed and emits only those values. Each integration receives a source payload tailored to its specific requirements.</p><p>The extract is smaller and faster for HCM to generate, while downstream processing is simpler because systems receive only the attributes that changed.</p><p>On the other hand, this creates a direct dependency between the HCM extract definition and the downstream system.</p><p>Our HCM experts chose BLOCK_OLD extraction mode. In that mode, Oracle HCM emits both current and previous values for much of the HCM employee model.</p><p>On top of that, they decided to create <strong>a generic HCM extract definition</strong> that could be reused for multiple integrations.</p><p>HCM extracts using BLOCK_OLD are much larger because they always contain all fields and their _OLD counterparts. They take longer to generate, and the downstream processing is more complex because it has to detect changes in the middleware.</p><p>The main reason for choosing this approach was the stronger contract it created between HCM and the middleware.</p><h3>The Constraint That Shaped the Architecture: Unpredictable Scale</h3><p>The primary constraint was the size and unpredictability of the HCM extracts.</p><p>We used Oracle HCM’s BLOCK_OLD extraction mode. The extract included both current and previous values for a large portion of the HCM employee model.</p><p>The number of employees per run was unpredictable. A valid extract could contain:</p><ul><li>No employees at all.</li><li>A handful of changed employees.</li><li>Thousands of employees during normal processing.</li><li>The entire population.</li></ul><p>Consequently, the downloaded payload could range from a few kilobytes to several gigabytes. The inbound flow had to handle this entire range.</p><p>This scale made it impractical to treat the entire extract as a single integration message.</p><p>A multi-gigabyte document shouldn&#39;t be copied repeatedly through an ESB pipeline, kept in memory, or retried from the start because of a single employee failure.</p><p>Instead, download it and persist it as a source container, then parse and split it into manageable units of work.</p><p>Let’s plug a splitter into the inbound flow and define a bounded unit of work.</p><h3>Parsing and Splitting HCM Extracts</h3><p>A full HCM extract may contain thousands of employee records.</p><p>Loading it into memory and processing it synchronously leads to predictable issues:</p><ul><li>Memory usage grows with the file.</li><li>One invalid employee can cause the entire run to fail.</li><li>Work cannot be distributed across consumers.</li><li>Retrying repeats already completed work.</li><li>Long transactions are difficult to observe and recover.</li></ul><p>Instead, the inbound flow parses the extract and splits it at a meaningful business boundary, typically the employee block defined by the extract schema.</p><p>The important design decision is not the physical size of each fragment, but choosing a business boundary that can be processed, retried, audited, and recovered independently.</p><p>In the implemented design, a splitter created <strong>a single, independently processable ESB message</strong> for each employee.</p><p>This employee-level boundary constrained each message to one employee and made processing, auditing, failure isolation, and recovery consistently granular.</p><p>The splitter <strong>streamed the extract</strong> and emitted employee messages incrementally, so the complete document never had to be loaded into memory.</p><p>We used multiple HCM extract versions. The inbound flow therefore had to select the correct splitter for each extract version.</p><p>Let’s extend the integration configuration to include the splitter version.</p><pre>&lt;integration&gt;<br>    &lt;id&gt;A&lt;/id&gt;<br><br>    &lt;!-- hcm extract parameters --&gt;<br>    &lt;hcmExtract&gt;<br>      &lt;id&gt;HCM_EMPLOYEE_EXTRACT&lt;/id&gt;<br>      &lt;country&gt;XY&lt;/country&gt;<br>      &lt;effectiveDate&gt;XYZ&lt;/effectiveDate&gt;<br>      &lt;!-- other parameters --&gt;<br>    &lt;/hcmExtract&gt;<br><br>    &lt;!-- splitter version to use for the extract --&gt;<br>    &lt;splitter&gt;1.0&lt;/splitter&gt;<br>  &lt;/integration&gt;</pre><p>In this architecture, the benefits of bounded work outweighed the added coordination.</p><blockquote>In this design, one ESB message corresponds to one employee.</blockquote><p>Once you split the extract into individual messages, multiple consumers running on separate runtime instances can process them in parallel.</p><p>However, this distributed model requires additional tracking and coordination: you must track each message and consider the extract complete only after every message has been processed.</p><p>This trade-off is worthwhile for the scalability and reliability it provides.</p><h3>A Sample ESB Message</h3><p>Each ESB message had two main parts: a header containing identifiers, routing information, and processing metadata, and a body containing the HCM employee extract segment.</p><p>The splitter created one such message for each employee block in an HCM extract.</p><p>A simplified representation of an ESB message:</p><pre>&lt;message&gt;<br>  &lt;header&gt;<br>    &lt;integration&gt;A&lt;/integration&gt;<br>    &lt;messageType&gt;Extract&lt;/messageType&gt;<br>    &lt;session&gt;398c41fb-dfe3-49d2-ab78-027161125bf9&lt;/session&gt;<br>    &lt;personNumber&gt;12345678&lt;/personNumber&gt;<br>    &lt;timestamp&gt;2026-05-24T11:48:14Z&lt;/timestamp&gt;<br>    &lt;!-- other metadata omitted --&gt;<br>  &lt;/header&gt;<br>  &lt;body&gt;EMPLOYEE_XML_FRAGMENT&lt;/body&gt;<br>&lt;/message&gt;</pre><p>The fields had distinct architectural roles:</p><ul><li>integration: integration ID used for configuration lookup.</li><li>messageType: used for routing within the ESB.</li><li>session: generated at the beginning of the inbound flow by the middleware as the unique end-to-end identifier for the extract. The same value correlated audit records across extract submission and download, inbound splitting, employee processing, merging, and outbound delivery.</li><li>personNumber: the secondary audit identifier. It made it possible to search for an employee in audit logs across all integrations (payroll, time and attendance, and others).</li><li>timestamp: recorded when the inbound flow published the message.</li><li>body: contained the HCM extract fragment for the individual employee.</li></ul><p>At this point, we successfully retrieved the HCM extract, parsed it, and split it into individual ESB messages. Each message contained a single employee block and was ready for downstream processing.</p><p>Let’s decouple the inbound flow from the downstream integrations and make it fully asynchronous.</p><h3>Let’s Add Some Plumbing: Inbound Queues</h3><p>I introduced asynchronous queues between the inbound flow and each integration.</p><p>Integrations subscribed to their own queues, decoupling extract ingestion from downstream processing early in the pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8-RND8Ax_txXdR5eNBG1xw.png" /></figure><p>This allowed each integration to scale and retry independently, while isolating most processing failures and backlogs from other integrations.</p><p>The queues also buffered extract production and downstream consumption, absorbing temporary processing backlogs without blocking the inbound flow.</p><p>We need to extend the integration configuration to include the inbound queue name. The inbound flow will read this value and publish the employee messages to the correct queue.</p><pre>&lt;integration&gt;<br>    &lt;id&gt;A&lt;/id&gt;<br><br>    &lt;!-- hcm extract parameters --&gt;<br>    &lt;hcmExtract&gt;<br>      &lt;id&gt;HCM_EMPLOYEE_EXTRACT&lt;/id&gt;<br>      &lt;mode&gt;BLOCK_OLD&lt;/mode&gt;<br>      &lt;country&gt;XYZ&lt;/country&gt;<br>      &lt;effectiveDate&gt;XYZ&lt;/effectiveDate&gt;<br>      &lt;!-- other parameters --&gt;<br>    &lt;/hcmExtract&gt;<br><br>    &lt;!-- splitter version to use for the extract --&gt;<br>    &lt;splitter&gt;1.0&lt;/splitter&gt;<br>    &lt;!-- inbound queue for the integration --&gt;<br>    &lt;inboundQueue&gt;Inbound_A&lt;/inboundQueue&gt;<br>  &lt;/integration&gt;</pre><p>At this point, we had addressed the three original issues: orchestration was shared, large extracts were divided into bounded units of work, and downstream processing was asynchronous and isolated.</p><h3>The “One Integration, One Inbound Queue” Principle</h3><p>This principle was the most important architectural decision. It shaped the inbound flow design, message routing and isolation, fault handling, and the platform’s overall scalability. It also resulted from several iterations and trade-off evaluations.</p><p>I still have flipcharts where I sketched the architecture and its alternatives, trying to find the golden rule.</p><p>I could have used a single inbound queue for all integrations, but that would have created shared contention and a shared failure domain. A backlog or misbehaving integration could have affected every other integration using the same queue.</p><p>That evaluation led to the <strong>one-integration-one-inbound-queue</strong> principle. It turned out to be the best decision I could have made for the platform’s long-term maintainability, scalability, and reliability.</p><p>Now it looks obvious, but it was not at the time. I had to evaluate several alternatives and consider the trade-offs before arriving at this principle.</p><p>If you are designing a similar platform and don’t know which systems you&#39;ll add in the future, I recommend considering this principle early in the design phase. It will save you a lot of headaches down the road.</p><blockquote>Divert traffic early in the pipeline, and isolate each integration via an inbound queue.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EUV7h-VnfEyDmLH91qXBpA.png" /></figure><p>Later, you can handle routing, transformation, and delivery details without worrying about cross-integration interference.</p><p>Queues also create a clear ownership boundary. The common inbound flow owns extraction, parsing, and publication.</p><p>The integration consumer owns transformation, mapping, and delivery.</p><h3>Multiple Downstream Systems per Integration</h3><p>The same isolation principle also solved a second routing problem: one country-level integration sometimes served multiple downstream systems.</p><p>In this architecture, a content-based router directed messages to the appropriate inbound queues.</p><p>The following diagram shows Integration E routing to multiple downstream systems via separate inbound queues.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PUddrxWQc8R-TQYd75k5WQ.png" /></figure><p>This allowed one country-level flow to fan out to multiple downstream systems without duplicating the HCM extract or the country-level inbound flow. Each destination retained its own queue and isolated package.</p><p>Each system had its own implementation package. This allowed for independent processing and deployment.</p><p>We could deploy a new version of the payroll integration without affecting other systems, and vice versa.</p><p>On top of that, we could gradually enable the downstream systems. For example, we could enable the payroll integration first, and then move the other integrations into production later.</p><p>We have covered the inbound flow, the splitting of HCM extracts into ESB messages, and routing to multiple integrations via inbound queues.</p><p>Now let’s take a closer look at the data models used throughout the payroll integrations. On one hand, we have the Oracle HCM extract model. On the other hand, we have the provider-specific models.</p><p>We need to bridge the gap between these two models.</p><h3>The Canonical Data Model</h3><p>Oracle HCM and payroll providers describe the same employee differently. Mapping the Oracle schema directly to every provider duplicates HCM interpretation logic and tightly couples every destination to the source.</p><p>A canonical data model (CDM) introduces a stable internal contract:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-Eq1r8TUaL7u88NrTYBDKQ.png" /></figure><p>The CDM was not a generic enterprise employee model imposed independently of the source. It was closer to a domain model: its structure evolved naturally from the Oracle HCM extract hierarchy.</p><p>This boundary reduces coupling in both directions. An Oracle extract change is absorbed primarily by the shared HCM-to-CDM transformation. A provider-format change is handled in that provider’s mapping.</p><p>I did not design the Oracle HCM model or the CDM model. The HCM domain experts provided those definitions.</p><p>My responsibility was technical stewardship of the CDM within the integration layer: maintaining and versioning its XSDs, updating transformations and dependent artifacts, and ensuring integrations stayed aligned with the latest approved HCM and CDM changes.</p><h3>Using XSLT as the Transformation Engine</h3><p>XSLT was the core transformation technology because both the HCM extract and the canonical representation were XML.</p><p>Each ESB message passed through the following stages:</p><ol><li>ESB message</li><li>XSLT inbound filter</li><li>XSLT HCM-to-CDM transformation</li><li>CDM message</li></ol><p>The <strong>XSLT inbound filter</strong> acted as an allowlist. It retained only fields required by the integration, without changing their values or the HCM structure. Besides reducing payload size, this created a data-minimization boundary: new Oracle fields did not automatically flow to a payroll provider.</p><p>The <strong>XSLT HCM-to-CDM transformation</strong> converted the filtered HCM hierarchy into the CDM hierarchy.</p><p>From then on, all payroll integrations worked exclusively with the canonical CDM messages.</p><p>A common data model as the central representation allowed all payroll integrations to use a consistent structure, reducing the complexity of handling multiple HCM-specific formats.</p><p>Common lifecycle events, such as hires, terminations, rehires, and updates, could be implemented consistently across all payroll integrations.</p><h3>The Outbound Flow</h3><p>The outbound design followed two patterns, selected by the destination contract.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gznAbM7hoUQev439gWN56Q.png" /></figure><h3>File-Based Delivery</h3><p>In this design, file-based integrations used one inbound queue and no outbound message queue. Parallel processors converted each employee ESB message into a provider-ready fragment and wrote it to the extract’s staging area.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VCmSWBb6s3fwDLW_54G1cA.png" /></figure><p>After splitting the extract into ESB messages, you need to track completion of all messages before merging them into one or more files.</p><p>In this design, an aggregator pattern collected the fragments and assembled them into the final file(s).</p><p>The completion tracker compared expected messages with completed messages. Once it reached the expected count, the tracker issued a command message notifying the aggregator to assemble the staged fragments into one or more files.</p><p>The aggregator was provider-aware only for file assembly: some providers expected a single consolidated file, while others required multiple files divided by record type or other delivery rules. Once assembled, every file intended for an external system was PGP-encrypted.</p><h3>Message-Based Delivery</h3><p>For message-based integrations delivered through request/response protocols, I introduced <strong>a separate outbound queue</strong>.</p><p>The integration-specific processing package detected business events, mapped the CDM message to the provider format, and published the final target payload to the outbound queue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0KpXgUqPdI2WKExURzOVHw.png" /></figure><p>This design had two main reasons. First, it decoupled the integration-specific processing from the delivery mechanism. Second, it allowed the outbound flow to retry transient failures without reprocessing the CDM message.</p><p>Outbound messages were <strong>immutable</strong>. The integration-specific processing package created the final target payload and published it to the outbound queue. The sender did not modify the payload.</p><p>Each hire, termination, address change, assignment change, or other detected change became its own outbound ESB message.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZWZEsX2tovSFWpcKEJ4qLg.png" /></figure><p>The outbound consumer was intentionally simple. It did not interpret HCM data, apply business rules, or modify the payload.</p><p>Its responsibilities were limited to transport concerns: establishing the connection, authenticating, sending the payload, interpreting the technical response, retrying transient failures, and recording the audit result.</p><p>If delivery failed, the same immutable target message could be retried per policy.</p><p>Let’s wrap up the architecture overview and summarize the key points.</p><h3>High-Level Architecture Overview</h3><p>The following diagram illustrates the integration platform&#39;s high-level architecture, including the inbound flow, canonical data model, integration-specific processing, and outbound delivery.</p><p>Integration A represents a file-based integration, while Integration B represents a message-based integration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q_dF73D-1vUPlAgmC-1TnQ.png" /></figure><p>The common inbound flow handled extract ingestion and published employee-level ESB messages to the integration-specific inbound queues.</p><p>XSLT filtering reduced the payload to the fields each integration required, and XSLT transformation converted the filtered HCM extract into the canonical data model.</p><p>Integration-specific logic processed CDM messages.</p><p>For file-based integrations (Integration A), the outbound flow aggregated file fragments into file(s), PGP-encrypted file(s), and delivered the file(s) to the target system.</p><p>For message-based integrations (Integration B), the integration logic published the final target payload to an outbound queue, which a transport-only sender consumed and delivered to the target system.</p><h3>Integration-Specific Implementations</h3><p>Here are a few examples of the integrations I implemented, each with its own requirements and challenges.</p><h3>First Integration: Inbound Filtering and Canonical Delivery</h3><p>The first integration I implemented was file-based.</p><p>I developed the common inbound flow, message routing via queues, <a href="https://medium.com/@stanislav.tvaruzek/filtering-xml-messages-using-xslt-452b46884b87">XSLT filtering</a> and transformation to the canonical data model, and the outbound flow for file aggregation, encryption, and delivery.</p><p>It was an important milestone because it validated the architecture and proved that the design could handle large HCM extracts, split them into manageable units, and deliver them to a downstream system.</p><p>It served as a proof-of-concept implementation for the entire processing pipeline and as a blueprint for all file-based integrations that followed.</p><h3>Second Integration: BLOCK_OLD and Event Detection</h3><p>The second integration I implemented was the first to require event detection.</p><h4>BLOCK_OLD and Implications for Middleware</h4><p>BLOCK_OLD shifted an important responsibility from HCM to the integration layer. The middleware had to compare current and previous values across many parts of the canonical message and translate those differences into business events such as address changes, working-hours changes, bank-account changes, and assignment changes.</p><h4>Trade-Offs Between the Approaches</h4><p>Neither extraction strategy is universally superior. Each places complexity in different parts of the processing pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xH5qcHfsv3oiBu89LspB5g.png" /></figure><p>CHANGES_ONLY reduces message size and lets HCM own change detection, but it can multiply the number of source extract definitions as destinations diverge.</p><p>BLOCK_OLD creates very large messages and shifts comparison complexity into middleware. Combined with a generic extract definition, BLOCK_OLD provided a stable, reusable source contract and made event logic explicit, testable, and configurable in the integration layer.</p><h4>Event Detection and Business Rules</h4><p>For each business event, I needed to define which fields in the CDM message could trigger it.</p><p>I made these rules configuration-driven. Each event declared the CDM detail segments it observed and the fields that could trigger it.</p><p>The middleware loaded this configuration, located each field and its _OLD counterpart, and compared their values. If at least one configured pair differed, it emitted the corresponding event. An event could monitor fields in several nested detail segments, which was important for complex business rules.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xXGq5AJl0yJsA5eIGCSpNw.png" /></figure><p>This separated event definitions from the comparison code. Adding a field to an address-change rule was a configuration change and did not require code changes.</p><p>The integration could listen for changes in multiple segments of the CDM message and emit a single event if any configured field changed.</p><p>This was another blueprint for all message-based integrations that followed. We reused the same generic comparison logic. Only the configuration changed.</p><h3>Extending the Inbound Flow Beyond Payroll</h3><p>We had a few integrations that synced organization data, and we wanted to avoid creating a separate inbound flow for each extract type. I extended the common flow to handle these cases, and migrated integrations to the new design.</p><p>I’m not going into much detail here, but for non-payroll extracts, the inbound flow issued a command message to a registered handler. The handler could then process the extract as required.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/524/1*0p8jQTILj70sKanN8DZAog.png" /></figure><p>The standard branch handled employee-oriented payroll extracts by splitting the file into independently processable employee-level ESB messages and publishing them to the inbound queue.</p><p>For non-payroll extracts, the flow sent a command message to a registered handler. The handler could then process the extract as required.</p><p>This is the strategy pattern applied at platform scale: the stable lifecycle remains centralized while content-specific processing is delegated.</p><h3>Testing and Quality Assurance</h3><p>Testing these integrations presented several challenges because of the data&#39;s complexity and variability. As we were dealing with massive XML structures, transforming them into the CDM format required validation to ensure both structural and semantic correctness.</p><h3>XSLT Transformation Testing</h3><p>XSLT transformation testing focused on verifying that the XSLT stylesheet correctly transformed the HCM employee fragment into the CDM format.</p><p>Because the transformation from HCM to CDM was the core part of the flow, ensuring its correctness was critical.</p><p>I created dedicated tests for every supported HCM extract version and its corresponding stylesheet to ensure accurate transformation.</p><p>Tests used representative HCM XML input fixtures for different HCM extract versions. Each fixture passed through the HCM-to-CDM transformation. The resulting CDM XML was validated against the CDM XSD and compared with the expected CDM message.</p><p>The test pattern was:</p><ol><li>HCM XML input fixture</li><li>Apply XSLT transformation</li><li>Validate resulting CDM XML against CDM XSD</li><li>Compare resulting CDM XML with expected CDM message</li></ol><h3>Integration Testing</h3><p>Testing every possible combination of values in HCM extracts is impossible, so I focused on the most important scenarios and edge cases.</p><p>I used automated tests extensively to ensure integrations produced the expected outputs.</p><p>The test pattern was:</p><ol><li>CDM input fixture</li><li>Call mapping service</li><li>Compare actual output with expected output</li><li>Validate against target schema</li></ol><p>Preparing representative fixtures required significant effort, but they became durable regression assets and reduced defects later in the lifecycle.</p><p>Every defect found during UAT was covered by a regression test. The same approach was applied to any defects discovered during production hypercare.</p><p>This ensured defects were systematically addressed and significantly reduced regression risk throughout the development lifecycle.</p><h3>Tracking and Audit Logging</h3><p>No single synchronous request spans the platform, so consistent correlation is essential. I used a two-level audit model:</p><ul><li>The middleware-generated session was the primary correlation ID. It identified one extract and followed it through inbound processing, every employee message, aggregation, and outbound delivery.</li><li>personNumber was the secondary correlation key. Searching for it returned the transactions processed for one employee across multiple integrations (payroll, time and attendance, etc.).</li></ul><p>Together they answered two important operational questions: “What happened to this extract?” and “What happened to this employee?”</p><h3>The Cost of Scalability</h3><p>This scalability came with additional coordination. Once the platform divided an extract into independent messages, it also had to track completion, manage correlation, and aggregate outputs where required.</p><p>It also introduced more infrastructure to operate, including durable queues, persistent state, and temporary staging for file-based integrations.</p><p>The design traded local simplicity for bounded processing, failure isolation, and recoverability at scale.</p><h3>Key Takeaways</h3><ul><li><strong>Scalable, Reliable Architecture</strong>: Asynchronous queues and a canonical data model enabled robust, global payroll integrations that adapt to changing requirements and unpredictable scale.</li><li><strong>Reusable Patterns</strong>: Standardizing inbound flows and leveraging configuration-driven integrations minimized complexity and improved maintainability.</li><li><strong>Efficient Processing</strong>: Splitting large employee extracts into smaller messages improved processing efficiency and system responsiveness.</li><li><strong>Quality Assurance</strong>: Automated testing and representative regression fixtures supported high quality throughout the development lifecycle.</li><li><strong>Operational Excellence</strong>: End-to-end audit logging made production troubleshooting and correlation significantly easier.</li><li><strong>Real-World Lessons</strong>: Embracing iterative improvements and learning from both successes and failures led to better design decisions and a resilient solution.</li><li><strong>Blueprint for Others</strong>: The principles, patterns, and lessons shared here provide a practical starting point for anyone architecting HCM extracts or large-scale payroll integrations.</li></ul><h3>Conclusion</h3><p>Looking back, the most consequential decisions were introducing queues, stopping treating a complete extract as one in-memory unit of work, and centralizing the inbound flow.</p><p>The difficult part was never simply transforming a single XML document into a provider file. It was designing — and then personally building and operating — a platform that could accept anything from an empty extract to several gigabytes of employee data, feed many independently evolving integrations, and remain recoverable in the event of partial failure.</p><p>The resulting architecture divided the work into clear responsibilities:</p><ul><li>A dedicated queue provides processing isolation for each integration.</li><li>Shared inbound orchestration submits, polls, downloads, and decrypts any configured extract.</li><li>Large employee extracts are split into independently processable employee-level messages.</li><li>XSLT filtering and HCM-to-CDM transformation create explicit, testable XML boundaries.</li><li>The CDM decouples the shared employee structure from destination-specific mapping.</li><li>Outbound flows support both transactional delivery and reliable file aggregation.</li></ul><p>The result is more than just a set of payroll interfaces. It is an integration platform that can add destinations, handle failures, and update its source and target contracts without redesigning the whole flow.</p><p>Just as importantly, its mappings and event rules are backed by executable examples, so reliability comes from repeatable engineering rather than mere confidence.</p><p>I designed and evolved this architecture over four years, from the first country implementation to a global rollout supported by a growing development team.</p><p>It was not only a design exercise: the architecture evolved through years of production operation, changing HCM schemas, new downstream systems, and lessons learned from real operational failures.</p><p>I wrote this article to share the architectural patterns, trade-offs, and operational lessons that proved most valuable.</p><p>I hope that some of the information shared here will help you in your own integration projects.</p><h3>Let’s Connect</h3><p>If you are working with webMethods, Oracle HCM, large-file processing, asynchronous integration architectures, or complex enterprise integration challenges, I would be happy to connect and exchange ideas.</p><p>Connect with me on <a href="https://www.linkedin.com/in/tvaruzek/">LinkedIn</a> to continue the conversation, share your challenges, or explore ways we might work together.</p><h3>Glossary</h3><p>BLOCK_OLD — An HCM extraction mode used here that carries current and previous values for middleware comparison.</p><p>CDM — A standardized and consistent data format used as an internal contract between systems, helping decouple sources and destinations.</p><p>CHANGES_ONLY — An HCM extraction strategy in which HCM emits the attributes it identifies as changed.</p><p>ESB — Enterprise Service Bus.</p><p>ESB Message — A structured message that flows through the ESB, containing information about an event or transaction.</p><p>HCM — Human Capital Management. Refers to systems and processes that manage an organization’s workforce. In this article, it refers solely to Oracle HCM Cloud.</p><p>HCM extract — A structured data export from Oracle HCM Cloud, containing employee information and related attributes. In this article, it refers to the XML-based extracts used for payroll and non-payroll integrations.</p><p>PGP — An encryption mechanism used to secure files or messages, especially for transferring sensitive data.</p><p>SFTP — A secure protocol for transferring files between systems over a network.</p><p>SOAP — XML-based web service messaging protocol.</p><p>webMethods — Enterprise integration platform used to implement processing services, packages, messaging flows, scheduling, and operational tooling.</p><p>XSD — XML Schema Definition.</p><p>XSLT — A language for transforming XML documents into other XML documents or different formats, commonly used for data mapping and transformation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9199317c080b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Filtering XML Messages Using XSLT]]></title>
            <link>https://medium.com/@stanislav.tvaruzek/filtering-xml-messages-using-xslt-452b46884b87?source=rss-195a54f44981------2</link>
            <guid isPermaLink="false">https://medium.com/p/452b46884b87</guid>
            <category><![CDATA[hcm-cloud]]></category>
            <category><![CDATA[xslt]]></category>
            <category><![CDATA[oracle-hcm]]></category>
            <category><![CDATA[xml]]></category>
            <category><![CDATA[webmethods-integration]]></category>
            <dc:creator><![CDATA[Stanislav Tvarůžek]]></dc:creator>
            <pubDate>Thu, 27 Jul 2023 15:08:49 GMT</pubDate>
            <atom:updated>2023-07-27T15:08:49.891Z</atom:updated>
            <content:encoded><![CDATA[<p>How to filter out unwanted XML elements from XML messages.</p><h3>Introduction</h3><p>I want to share an elegant technique for filtering XML messages in this article.</p><p>I will demonstrate how to create a future-proof solution to handle structural changes in XML messages.</p><p>This technique helps with XML messages, where the structure might change over time.</p><p>Target audience: developers, software engineers, and architects.</p><h3>Background</h3><p>Back in the day, I was tasked to devise a solution to filter XML messages from Oracle Human Capital Management (HCM). The incoming messages were in XML format. In HCM terminology, they are called HCM extracts.</p><p>The HCM extracts were processed by middleware (webMethods Integration Platform) and sent to various downstream systems. We have here one-to-many integration where one source system feeds multiple downstream systems.</p><figure><img alt="High-Level Overview" src="https://cdn-images-1.medium.com/max/587/1*6GwigvT5iGwrJkt3lXwRvw.png" /><figcaption>High-Level Overview</figcaption></figure><p>Terms message, XML message, HCM message, and HCM extract are used interchangeably in this article and represent information in XML format.</p><h3>Main Requirement</h3><p>The main requirement is that each downstream system must receive only an exact subset of XML elements it needs and nothing else.</p><h3>Let&#39;s Get Technical</h3><p>Let&#39;s say you have a message like this:</p><pre>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;<br>&lt;ns0:Employee xmlns:ns0=&quot;https://www.example.com/employee&quot;&gt;<br>    &lt;ns0:Name&gt;John&lt;/ns0:Name&gt;<br>    &lt;ns0:Age&gt;30&lt;/ns0:Age&gt;<br>    &lt;ns0:Address&gt;<br>        &lt;ns0:Street&gt;123 Main Street&lt;/ns0:Street&gt;<br>        &lt;ns0:City&gt;San Francisco&lt;/ns0:City&gt;<br>        &lt;ns0:State&gt;CA&lt;/ns0:State&gt;<br>        &lt;ns0:Zip&gt;94105&lt;/ns0:Zip&gt;<br>    &lt;/ns0:Address&gt;<br>    &lt;ns0:Phone&gt;<br>        &lt;ns0:Home&gt;111-111-1111&lt;/ns0:Home&gt;<br>        &lt;ns0:Mobile&gt;222-222-2222&lt;/ns0:Mobile&gt;<br>    &lt;/ns0:Phone&gt;<br>&lt;/ns0:Employee&gt;</pre><p>For a downstream System A you want to keep only &lt;ns0:Name&gt; and &lt;ns0:Zip&gt; elements. The result should look like this:</p><pre>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;<br>&lt;ns0:Employee xmlns:ns0=&quot;https://www.example.com/employee&quot;&gt;<br>    &lt;ns0:Name&gt;John&lt;/ns0:Name&gt;<br>    &lt;ns0:Address&gt;<br>        &lt;ns0:Zip&gt;94105&lt;/ns0:Zip&gt;<br>    &lt;/ns0:Address&gt;<br>&lt;/ns0:Employee&gt;</pre><p>System B requires only &lt;ns0:Name&gt; and &lt;ns0:Mobile&gt; elements. The result should look like this:</p><pre>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;<br>&lt;ns0:Employee xmlns:ns0=&quot;https://www.example.com/employee&quot;&gt;<br>    &lt;ns0:Name&gt;John&lt;/ns0:Name&gt;<br>    &lt;ns0:Phone&gt;<br>        &lt;ns0:Mobile&gt;222-222-2222&lt;/ns0:Mobile&gt;<br>    &lt;/ns0:Phone&gt;<br>&lt;/ns0:Employee&gt;</pre><p>At first sight, this is straightforward. You store somewhere a list of XML elements you want to keep for each downstream system. You traverse incoming messages and remove the elements not present in the list.</p><p>This is a perfectly valid approach. Doable in any programming language. But this approach has a few drawbacks. The main one is that you must write the code, test it and debug it. And this will be, for sure, an error-prone and time-consuming process.</p><h3>The Catch</h3><p>The catch is that the set of XML elements in HCM extracts can change over time. HCM extracts define which employee attributes are extracted from Oracle HCM.</p><p>An HCM extract is a structured list of employee attributes used to extract data from Oracle HCM. Something like a report where you can define which employee attributes you want to extract from HCM.</p><p>Developers of HCM extracts <strong>can add new XML elements, or they can add brand-new levels</strong> to the extract definition.</p><p>On the one hand, you must process inbound messages without control over the structure. On the other hand, you have downstream systems that require messages precisely in the structure they expect.</p><p>Let&#39;s say that the extract definition has changed and &lt;ns0:Surname&gt; and &lt;ns0:Email&gt; were added. Sample message:</p><pre>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;<br>&lt;ns0:Employee xmlns:ns0=&quot;https://www.example.com/employee&quot;&gt;<br>    &lt;ns0:Name&gt;John&lt;/ns0:Name&gt;<br>    &lt;ns0:Surname&gt;Doe&lt;/ns0:Surname&gt;<br>    &lt;ns0:Age&gt;30&lt;/ns0:Age&gt;<br>    &lt;ns0:Address&gt;<br>        &lt;ns0:Street&gt;123 Main Street&lt;/ns0:Street&gt;<br>        &lt;ns0:City&gt;San Francisco&lt;/ns0:City&gt;<br>        &lt;ns0:State&gt;CA&lt;/ns0:State&gt;<br>        &lt;ns0:Zip&gt;94105&lt;/ns0:Zip&gt;<br>    &lt;/ns0:Address&gt;<br>    &lt;ns0:Email&gt;<br>        &lt;ns0:Work&gt;john.doe@acme.com&lt;/ns0:Work&gt;<br>        &lt;ns0:Personal&gt;john.doe@yahoo.com&lt;/ns0:Personal&gt;<br>    &lt;/ns0:Email&gt;<br>    &lt;ns0:Phone&gt;<br>        &lt;ns0:Home&gt;111-111-1111&lt;/ns0:Home&gt;<br>        &lt;ns0:Mobile&gt;222-222-2222&lt;/ns0:Mobile&gt;<br>    &lt;/ns0:Phone&gt;<br>&lt;/ns0:Employee&gt;</pre><p>System A must still receive this message:</p><pre>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;<br>&lt;ns0:Employee xmlns:ns0=&quot;https://www.example.com/employee&quot;&gt;<br>    &lt;ns0:Name&gt;John&lt;/ns0:Name&gt;<br>    &lt;ns0:Address&gt;<br>        &lt;ns0:Zip&gt;94105&lt;/ns0:Zip&gt;<br>    &lt;/ns0:Address&gt;<br>&lt;/ns0:Employee&gt;</pre><p>Furthermore, an extract contains only data entered in HCM. For example, the element will be empty if an employee has no phone.</p><pre>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;<br>&lt;ns0:Employee xmlns:ns0=&quot;https://www.example.com/employee&quot;&gt;<br>    &lt;ns0:Name&gt;John&lt;/ns0:Name&gt;<br>    &lt;ns0:Age&gt;30&lt;/ns0:Age&gt;<br>    &lt;ns0:Address&gt;<br>        &lt;ns0:Street&gt;123 Main Street&lt;/ns0:Street&gt;<br>        &lt;ns0:City&gt;San Francisco&lt;/ns0:City&gt;<br>        &lt;ns0:State&gt;CA&lt;/ns0:State&gt;<br>        &lt;ns0:Zip&gt;94105&lt;/ns0:Zip&gt;<br>    &lt;/ns0:Address&gt;<br>    &lt;ns0:Phone/&gt;<br>&lt;/ns0:Employee&gt;</pre><p>So I had to ensure that the solution was flexible enough to handle such structural changes and would be future-proof.</p><p>On top of that, I&#39;m a terrible programmer and a super-lazy guy.</p><blockquote><em>And at the end of the day, I am a software engineer, not a programmer.</em></blockquote><p>So I was looking for a solution requiring minimum coding and zero maintenance.</p><h3>XSLT To The Rescue?</h3><p>I took a few courses at my alma mater about XML, and I was aware of XSLT. The last time I used it was almost twenty years ago on my first integration project, and I wondered if XSLT could do what I wanted.</p><p>Moreover, I was limited to using XSLT 1.0 as the integration tool (webMethods Integration Platform) supports only XSLT 1.0 out of the box.</p><p>So I pulled up my sleeves and sought a solution to leverage XSLT 1.0 capabilities.</p><h3>Searching For a Solution</h3><p>I wasn&#39;t looking for a solution that would remove defined XML elements. Any change in the extract definition would require a change in the XSLT for each downstream system.</p><p>But on the contrary, I wanted to keep only the XML elements I wanted and eliminate the rest. No matter how the structure of a message changes, the XSLT should remain untouched.</p><blockquote><em>Zero-maintenance solution. That was the goal.</em></blockquote><p>After a few hours of Googling, I stumbled upon a <a href="https://stackoverflow.com/questions/5790031/xslt-how-to-keep-only-wanted-elements-from-xml">StackOverflow</a> question similar to my problem. The solution was to use the following XSLT template:</p><pre>&lt;xsl:stylesheet version=&quot;1.0&quot;<br>    xmlns:xsl=&quot;http://www.w3.org/1999/XSL/Transform&quot;<br>    xmlns:ns=&quot;some:ns&quot;&gt;<br>    &lt;xsl:output omit-xml-declaration=&quot;yes&quot; indent=&quot;yes&quot;/&gt;<br>    &lt;xsl:strip-space elements=&quot;*&quot;/&gt;<br><br>    &lt;ns:WhiteList&gt;<br>        &lt;name&gt;ns:currency&lt;/name&gt;<br>        &lt;name&gt;ns:currency_code3&lt;/name&gt;<br>    &lt;/ns:WhiteList&gt;<br><br>    &lt;xsl:template match=&quot;node()|@*&quot;&gt;<br>        &lt;xsl:copy&gt;<br>            &lt;xsl:apply-templates select=&quot;node()|@*&quot;/&gt;<br>        &lt;/xsl:copy&gt;<br>    &lt;/xsl:template&gt;<br><br>    &lt;xsl:template match=&quot;*[not(descendant-or-self::*[name()=document(&#39;&#39;)/*/ns:WhiteList/*])]&quot;/&gt;<br>&lt;/xsl:stylesheet&gt;</pre><p>Author: <a href="https://stackoverflow.com/users/36305/dimitre-novatchev">Dimitre Novatchev</a></p><p>And this, ladies and gentlemen, was the solution to my problem! I just had to modify elements in the WhiteList.</p><p>And the best part is that I did not have to write a single line of code. I just had to configure the integration tool to use this XSLT template.</p><p>Pure beauty!</p><h3>How Does It Work?</h3><p>As Dimitre explains in his answer, the identity transform template is used to copy all nodes and attributes. Then it uses the second template to remove all elements not defined in the WhiteList (while preserving the structure).</p><blockquote><em>Explanation:</em></blockquote><blockquote>The identity rule/template copies all nodes &quot;as-is&quot;.</blockquote><blockquote>The stylesheet contains a top-level &lt;ns:WhiteList&gt; element whose &lt;name&gt; children specify all white-listed element&#39;s names -- the elements that are to be preserved with their structural relationships in the document.</blockquote><blockquote>The second template is overriding the identity template. It doesn&#39;t process (deletes) any element that is not white-listed and has no descendant that is white-listed.`</blockquote><h3>Let&#39;s Try It Out</h3><p>Fire up your browser and go to <a href="http://xsltransform.net/">http://xsltransform.net/</a> or download the source code from <a href="https://github.com/stvaruzek/xslt-message-filter/tree/main/src">this repository</a> and run it locally.</p><p>To make it work, you need to use a modified template that does not use the self-reference (document(&#39;&#39;)/*/ns:WhiteList/*]). That does not work in the web implementation. But it will work for sure in any XSLT processor.</p><pre>&lt;xsl:stylesheet version=&quot;1.0&quot;<br>    xmlns:xsl=&quot;http://www.w3.org/1999/XSL/Transform&quot;<br>    xmlns:ns=&quot;some:ns&quot;&gt;<br>    &lt;xsl:output omit-xml-declaration=&quot;yes&quot; indent=&quot;yes&quot;/&gt;<br>    &lt;xsl:strip-space elements=&quot;*&quot;/&gt;<br>    &lt;xsl:template match=&quot;node()|@*&quot;&gt;<br>        &lt;xsl:copy&gt;<br>            &lt;xsl:apply-templates select=&quot;node()|@*&quot;/&gt;<br>        &lt;/xsl:copy&gt;<br>    &lt;/xsl:template&gt;<br>    &lt;xsl:template match=&quot;*[not(descendant-or-self::*[name()=(&#39;ns0:Name&#39;, &#39;ns0:Zip&#39;)])]&quot;/&gt;<br>&lt;/xsl:stylesheet&gt;</pre><p>Input XML:</p><pre>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;<br>&lt;ns0:Employee xmlns:ns0=&quot;https://www.example.com/employee&quot;&gt;<br>    &lt;ns0:Name&gt;John&lt;/ns0:Name&gt;<br>    &lt;ns0:Surname&gt;Doe&lt;/ns0:Surname&gt;<br>    &lt;ns0:Age&gt;30&lt;/ns0:Age&gt;<br>    &lt;ns0:Address&gt;<br>        &lt;ns0:Street&gt;123 Main Street&lt;/ns0:Street&gt;<br>        &lt;ns0:City&gt;San Francisco&lt;/ns0:City&gt;<br>        &lt;ns0:State&gt;CA&lt;/ns0:State&gt;<br>        &lt;ns0:Zip&gt;94105&lt;/ns0:Zip&gt;<br>    &lt;/ns0:Address&gt;<br>    &lt;ns0:Email&gt;<br>        &lt;ns0:Work&gt;john.doe@example.com&lt;/ns0:Work&gt;<br>        &lt;ns0:Personal&gt;jodo@yahoo.com&lt;/ns0:Personal&gt;<br>    &lt;/ns0:Email&gt;<br>    &lt;ns0:Phone&gt;<br>        &lt;ns0:Home&gt;111-111-1111&lt;/ns0:Home&gt;<br>        &lt;ns0:Mobile&gt;222-222-2222&lt;/ns0:Mobile&gt;<br>    &lt;/ns0:Phone&gt;<br>&lt;/ns0:Employee&gt;</pre><p>Output XML:</p><pre>&lt;ns0:Employee xmlns:ns0=&quot;https://www.example.com/employee&quot;&gt;<br>    &lt;ns0:Name&gt;John&lt;/ns0:Name&gt;<br>    &lt;ns0:Address&gt;<br>        &lt;ns0:Zip&gt;94105&lt;/ns0:Zip&gt;<br>    &lt;/ns0:Address&gt;<br>&lt;/ns0:Employee&gt;</pre><p>You can see that the output XML contains only the white-listed elements (&#39;ns0:Name&#39;, &#39;ns0:Zip&#39;), and the structure is preserved. &lt;ns0:Address&gt; is included in the output.</p><p>Try to change the whitelist elements or the input XML and see what happens.</p><h3>One Important Remark</h3><p>To bring this solution to fruition, you must ensure that the <strong>names of XML elements are unique</strong>.</p><p>Once I was confident that I could go forward with this solution in the middleware layer and I could guarantee that each downstream system would receive only the elements that it was expecting, I requested the developers of HCM extracts to ensure that the names of the elements in the HCM extract definition were unique. I had the luxury of defining the names of XML elements in the HCM extract, which is not always the case.</p><h3>Where to Use It?</h3><p>As Dimitre points out in the comments:</p><blockquote><em>“I am providing a more general solution that solves a whole class of such problems.“</em></blockquote><p>Message filtering is a widespread scenario in system integration. Sending back-and-forth messages where only a fraction of the data is needed is a bad practice.</p><h3>Content Filter</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/366/0*_ylYuLweFVkZ2zDK.gif" /><figcaption><a href="https://www.enterpriseintegrationpatterns.com/patterns/messaging/ContentFilter.html">Enterprise Integration Patterns</a></figcaption></figure><blockquote><em>“Content Filter removes unimportant data items from a message leaving only important items.”</em></blockquote><h3>XSLT Content Filter</h3><p>Using XSLT for XML content filtering is an exquisite solution. It is easy to implement and easy to maintain.</p><h3>XSLT Inbound Message Filter</h3><p>In the project that I was working on, I used this solution to filter inbound messages coming from Oracle HCM. For each downstream system, I created a separate XSLT template. I put it under version control and configured the integration tool to use the appropriate template for each downstream system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*GDOyyRPtmR5XTRJ9DdGZtQ.png" /><figcaption>XSLT Inbound Message Filter</figcaption></figure><p>It worked like a charm.</p><h3>XSLT Outbound Message Filter</h3><p>One of the downstream systems was SAP. We integrated using SAP RFC, and the middleware sent one or more SAP RFC messages (so-called events) per one inbound XML message to the SAP system. The message format was flat XML, key-value pairs.</p><p>Like:</p><pre>&lt;message&gt;<br>    &lt;BEGDA&gt;2019-01-01&lt;/BEGDA&gt;<br>    &lt;ENDDA&gt;2019-12-31&lt;/ENDDA&gt;<br>&lt;/message&gt;</pre><p>We had many so-called events. All events shared the same SAP RCF message structure, but depending on the event, some XML elements were required in the message, and some were not.</p><p>For example, an event A required the BEGDA and ENDDA elements, but an event B required only the BEGDA element.</p><p>And there were plenty of such events to handle. And the SAP RFC message had more than 100 fields. Classic mapping nightmare. Luckily, <strong>the mapping from the HCM message to the SAP RFC message was always the same for each XML element for all events.</strong></p><p>Initially, I created a mapping service for each event, but as the number of events increased, I quickly realized that was a recipe for disaster. I would have to maintain many mapping services. And if I wanted to change the mapping for a single XML element across all events (due to a change in the mapping specification), I would have to change it in many places, which could break the mapping logic like a house of cards.</p><p>Then I got enlightened. I could use the same technique for outbound message filtering!</p><p>Instead of having a mapping service for each event, I created <strong>a single service that always mapped all fields</strong> to the SAP RFC message. Then, depending on the event ID, I overrode the values with event-specific values.</p><p>And finally, depending on the event ID, I applied an XSLT template to filter out all elements not required by the SAP system for that particular event.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/1*ICmPuEVseq8IX_bEmsunfg.png" /><figcaption>XSLT Outbound Message Filter</figcaption></figure><p>What a beauty! The outbound message filter ensured that SAP received only the required fields per event, and I had to manage only one mapping service. Plus, cosmetic mapping changes per event.</p><p>More than a year in production and zero defects.</p><h3>Dynamic XSLT Filters</h3><p>I used a static XSLT template for each downstream system/event in the previous example. You can populate the WhiteListelement in the XSLT template dynamically based on the elements you want to keep.</p><p>For my use case, I did not need it as I prefer to have a static XSLT template under version control.</p><h3>Things to Watch Out For</h3><p>Currently, there are three versions of XSLT: XSLT 1.0, XSLT 2.0, and XSLT 3.0. XSLT 1.0 and XSLT 2.0 are W3C recommendations. XSLT 3.0 is a community standard. All major XSLT processors support XSLT 1.0. XSLT 2.0 is supported by most of them. A few of them support XSLT 3.0.</p><p>The main difference is that XSLT 1.0 and XSLT 2.0 use a document object model (DOM), while XSLT 3.0 uses streaming. Meaning that XSLT 1.0 and XSLT 2.0 processors load the whole XML message into memory first and then apply an XSLT transformation, and you can eventually run out of memory.</p><p><strong>If you have no control over the size of the message, you will hit an out-of-memory error sooner or later.</strong> I learned the hard way :) I might write a separate article about it.</p><p>The only XSLT option in such a case is XSLT 3.0, which is not widely supported. I played a bit with the Java implementation of XSLT 3.0 — Saxon-EE, which supports streaming and can handle huge messages (I tested it on HCM extracts up to 500 MB in size), but it is not free.</p><h3>JSON Filtering</h3><p>As XML and JSON are interchangeable, you can also use XSLT to filter JSON messages. You need to convert JSON to XML, filter it using XSLT, and convert it back to JSON. Sure, there are plenty of other ways to filter JSON messages, but this could be a way, especially if you deal with JSON messages, where JSON structure might change over time.</p><h3>XSLT in Browser</h3><p>All popular browsers support XSLT 1.0. You can use it to transform XML to HTML or vice versa or process XML messages in the browser. I put in <a href="https://github.com/stvaruzek/xslt-message-filter/tree/main/src">the repository</a> a simple example of how to use XSLT in the browser. Just open <a href="https://github.com/stvaruzek/xslt-message-filter/blob/main/src/index.html">index.html</a> in your browser and modify the <a href="https://github.com/stvaruzek/xslt-message-filter/blob/main/src/message.xml">message.xml</a> file or <a href="https://github.com/stvaruzek/xslt-message-filter/blob/main/src/filter.xslt">filter.xslt</a> file.</p><h3>Main Takeaways</h3><ul><li><strong>Be lazy, don&#39;t write code if you don&#39;t have to.</strong> If you can solve a problem without writing code, do it.</li><li>Instead of messing with XML parsers, research, and you might find an XSLT solution to your problem. And you will learn something new along the way.</li><li><strong>XSLT is language agnostic.</strong> You can develop XSLT transformations in your favorite language and IDE and use another language in your run-time environment.</li><li>When dealing with XML messages, XSLT is your first bet. Even though XSLT is an alien for many developers, it is a mighty hammer on XML.</li><li>All major browsers support XSLT 1.0</li><li><strong>XSLT 1.0 uses DOM</strong>, which is not very memory efficient</li><li>XSLT 2.0 uses XPath 2.0, which is very powerful</li><li><strong>XSLT 3.0 uses streaming</strong>, which is very memory efficient but not widely supported</li></ul><h3>Conclusion</h3><p>I have shown you how to leverage XSLT to filter XML messages without coding. XSLT is a language that could be more popular among developers. It is an alien language for many. But when dealing with XML, it should be your first bet.</p><p>In the context of the project I was working on, the XSLT content filter perfectly fits the overall solution design.</p><p>I used it again in the project to convert HCM extracts to canonical XML documents without using an XML parser.</p><p>On top of that, my fellow developers could understand the XSLT code and modify it if needed. Also, it was easy for business users to understand which XML elements were interfaced to downstream systems.</p><p><strong>Post Scriptum #1:</strong> XML examples in this article are simplified for clarity. Those who work with HCM extracts know what I am talking about :) HCM extracts can be colossal, having thousands of XML elements per single employee.</p><p><strong>Post Scriptum #2:</strong> I am not an XSLT expert. I am just an ordinary software engineer who had to deal with XML messages with changing structures and had to find a viable solution. There are many other ways to filter XML messages. I wanted to share my experience with you.</p><p>I encourage you to give XSLT a try.</p><p>Thank you for reading.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=452b46884b87" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Create Digital Membership Cards with QR Codes]]></title>
            <link>https://medium.com/@stanislav.tvaruzek/how-to-create-digital-membership-cards-with-qr-codes-b9c8429de92d?source=rss-195a54f44981------2</link>
            <guid isPermaLink="false">https://medium.com/p/b9c8429de92d</guid>
            <category><![CDATA[membership-cards]]></category>
            <category><![CDATA[anymerge]]></category>
            <category><![CDATA[qr-code]]></category>
            <category><![CDATA[gmail-mail-merge]]></category>
            <dc:creator><![CDATA[Stanislav Tvarůžek]]></dc:creator>
            <pubDate>Fri, 08 Jan 2021 13:59:43 GMT</pubDate>
            <atom:updated>2021-01-08T15:30:40.178Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D8oZ9oaAeDqNo3V7vfox7A.jpeg" /><figcaption>Digital Membership Card. Author <a href="http://sarnovskydesigners.sk/">Andrej Sarnovský</a>.</figcaption></figure><p><strong>DISCLAIMER:</strong> I’m the author of the <a href="http://bit.ly/3bl5z8E">AnyMerge Mail</a> add-on.</p><p>In <a href="https://medium.com/@stanislav.tvaruzek/gmail-mail-merge-with-anymerge-mail-9dad860cd7ce">the previous article</a>, I introduced the AnyMerge Mail add-on I created for Google Spreadsheets. In today’s post, I’m going to show you how you can use the add-on to <strong>create digital membership cards with QR codes within five minutes and literally for free</strong>.</p><h3>What is a Digital Membership Card?</h3><p>By a digital membership card in this article, I mean <strong>an email message with a QR code</strong>. No that digital card where you need an extra application to be installed on your device. Everyone has an email client installed on a smartphone.</p><h3>Why Digital Membership Cards?</h3><p>Instead of making physical cards that your members will lose or forget to bring along go green and send cards directly from Gmail. Nowadays almost everyone carries a smartphone, right? In case someone accidentally deleted an email with a membership card you can re-send it again.</p><h3>Creating Digital Membership Cards in 5 Minutes</h3><p>It is super simple. <a href="http://bit.ly/3bl5z8E">Just install the add-on.</a> Once you open the add-on, it will guide you in your first steps and it will configure a sample setup for you.</p><p>It will:</p><ul><li>create a sample membership cards sheet with data</li><li>create a sample membership card Gmail draft with merge fields</li><li>set up a mapping between the merge fields in the draft and columns in the sheet</li><li>set up an output configuration</li></ul><h3>Gmail Mail Merge — Sample Sheet</h3><p>Add-on will create a sample membership card sheet. We will be sending membership cards to the Simpsons family.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MG3tcglvBt_H1aW1w6qRMw.png" /><figcaption>Sample Sheet</figcaption></figure><h3>Gmail Mail Merge — Sample Email Template</h3><p>Add-on creates a sample Gmail template. The template contains <strong>merge fields</strong>. Each merge field is enclosed in double curly brackets. In the picture below there are two merge fields visible — <strong>{{Name}}</strong> and <strong>{{QR}}</strong><em> </em>and there are a few more below the membership card.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lmjzYHbDk1GpHyGfXwNhCA.png" /><figcaption>Sample Email Template</figcaption></figure><h3>Gmail Mail Merge — Sample Mapping</h3><p>Add-on sets up a sample mapping between the merge fields in the sample draft and the columns in the sample sheet. <strong>The mapping tells the add-on how to process each merge field</strong> — from which column to take values and how to merge them into the sample template. Add-on can merge data as text values, images, and QR codes. On top of that, the add-on can <strong>attach multiple Google Drive files as attachments</strong>.</p><p>In this case, the add-on maps the {{QR}} merge field to the Card ID column and configures the merge type as QR Code. Values from the Card ID column will be encoded into QR Codes and placed where the {{QR}} merge field is.</p><p>You can adjust the mapping when you go to <em>Sheet Configuration -&gt; Merge Fields</em>.</p><h3>Gmail Mail Merge — Create Membership Cards</h3><p>Keep the default options on the main screen and hit the <em>GENERATE </em>button. The add-on will go row by row and for each row, it will create a personalized membership card. And that&#39;s it! Check out how the cards look like on your smartphone.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FlBh4y69sYh8%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DlBh4y69sYh8&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FlBh4y69sYh8%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ab479f20234e243e46d3ffcaead415a6/href">https://medium.com/media/ab479f20234e243e46d3ffcaead415a6/href</a></iframe><h3>Conclusion</h3><p>Streamline the card creation process with the <a href="http://bit.ly/3bl5z8E">AnyMerge Mail</a> add-on. Create your own template and send cards with QR codes from Gmail. Use your existing tracking solution to scan the QR codes.</p><p>Whether you run a club, organize a sports event, a music event, or a conference, take advantage of digital cards with QR codes generated using the <a href="http://bit.ly/3bl5z8E">AnyMerge Mail</a> add-on. It is for free.</p><p>Happy merging!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b9c8429de92d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Gmail Mail Merge with AnyMerge Mail]]></title>
            <link>https://medium.com/@stanislav.tvaruzek/gmail-mail-merge-with-anymerge-mail-9dad860cd7ce?source=rss-195a54f44981------2</link>
            <guid isPermaLink="false">https://medium.com/p/9dad860cd7ce</guid>
            <category><![CDATA[mail-merge]]></category>
            <category><![CDATA[gmail]]></category>
            <category><![CDATA[google-docs]]></category>
            <dc:creator><![CDATA[Stanislav Tvarůžek]]></dc:creator>
            <pubDate>Mon, 16 Nov 2020 17:02:34 GMT</pubDate>
            <atom:updated>2020-11-16T17:02:34.388Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7LEdwnK3Cl5_-Aot" /><figcaption>Photo by <a href="https://unsplash.com/@liamtruong?utm_source=medium&amp;utm_medium=referral">Liam Truong</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Merge spreadsheet data into Gmail messages like a boss!</p><h3>INTRODUCTION</h3><p>I published recently my brand new add-on for Google Spreadsheets for Gmail mail merge called <a href="https://bit.ly/3lwQ93g">AnyMerge Mail</a> and I would like to bring it to your attention. If you are looking for a way to streamline the email creation process in Gmail, stay tuned.</p><h3>WHAT IS A MAIL MERGE ADD-ON?</h3><p>Simply put, a mail merge add-on adds additional functionality to Google Spreadsheets and automates mail creation process. It takes a Gmail draft template, this template contains special placeholders or merge fields, and the add-on replaces merge fields with values from a sheet. It goes row by row, takes the template, makes a copy and fills in data. Finally, it sends emails to recipients.</p><h3>MOTIVATION</h3><p>My motivation was to build <strong>a suite of add-ons that excel in one thing and that is reliability</strong>. My background is system integration and I built the add-ons from the ground to be as reliable as possible. There is a lot of logic under the hood to make sure the add-ons deliver results as promised.<strong> AnyMerge add-ons are meant to be used for heavy duty work.</strong> Whether you want to send hundreds of personalized emails with <a href="https://bit.ly/3lwQ93g">AnyMerge Mail</a> add-on or generate hundreds of Google Documents with <a href="https://bit.ly/35yobPa">AnyMerge Docs</a> add-on.</p><h3>NEW KID ON THE BLOCK</h3><p>AnyMerge Mail add-on for Google Spreadsheets is the new kind on the block. There are plenty of other mail merge add-ons yet AnyMerge Mail stands out and offers features other add-ons do not offer.</p><h3>NO DAILY SENDING LIMIT</h3><p>While other mail merge add-ons let you send just a fraction of your daily quota allowed by Google for free, <strong>AnyMerge Mail add-on does not have a daily sending limit at all</strong>. What only limits you is the type of Gmail account you use. With a business account you can send up to 1 500 personalized emails per day from the add-on and reach your audience.</p><h3>USE IT FOR FREE</h3><p>You can use the add-on absolutely free of charge. Well, there is a small catch when using the add-on for free — the add-on appends a link to each message you send. I am more than happy to see people using the add-on for free but spread the word. I put a lot of effort into developing the add-on.</p><h3>MERGE QR CODES AND IMAGES INTO GMAIL MESSAGES</h3><p>AnyMerge Mail does not do just simple text merge. <strong>You can do pretty complex stuff</strong> like merging multiple QR codes, images as well as attachments into Gmail messages</p><h3>ADD MULTIPLE ATTACHMENTS</h3><p>Furthermore, the add-on allows you to add multiple attachments from your Google drive and attach them to Gmail messages. So you can further automate your daily workflows. You can use AnyMerge Docs add-on to generate Google Documents first and then send them with AnyMerge Mail add-on as attachments.</p><p>For example, you can generate a bunch of invoices first with AnyMerge Docs add-on and then send them with AnyMerge Mail add-on. <strong>You can streamline the entire process and move from manual, error prone process to automated workflow.</strong></p><h3>TWO-PHASE MERGE</h3><p>One of the unique features of the add-on is so-called two-phase merge. The add-on can merge spreadsheet data into Gmail drafts only and you can send the drafts later. Why would this be useful?</p><p><strong>The two-phase merge is particularly handy for long running tasks</strong> where you would like to process hundreds of emails with very complex setup. Doing simple merge of text values into Gmail messages is fast. Once you start adding more complex stuff like QR codes or multiple attachments processing times will increase and you can easily end up with processing times over one minute for a single message.</p><p>With two-phase merge you can let the add-on create drafts only (phase one) first. Depending on how many emails you would like to process it might take several hours to complete. Just make sure your workstation will not get to sleep and keep an eye on the process. Or run the merge job overnight. Once the merge job is over and all those drafts are sitting in your drafts folder you can send them (phase two). Sending already created drafts is much faster.</p><p>Another great advantage is that <strong>there is no daily quota on how many drafts you can create</strong>. Literally, you can create thousands of drafts per day and send them later in smaller batches.</p><blockquote>Take advantage of two-phase merge when your merge setup is complex and takes a long time to finish. Generate drafts first and send them later.</blockquote><h3>NO EMAIL TRACKING</h3><p>The add-on lacks some features other add-ons do provide. Most prominently it is email tracking. If this is the top feature you are looking for, do not waste your time and look for other solutions. I am not going to implement this feature in the foreseeable future.</p><h3>PRIVACY MATTERS</h3><p>Once an add-on has access to your Gmail it can access your Gmail messages and there is a possibility of leaking private data. AnyMerge Mail add-on cannot make external requests. Meaning that there is no way the add-on can send your data to a third-party. When you install the add-on you need to grant permission and authorize the add-on. For a detailed explanation of what the add-on does please read the <a href="https://mail.anymerge.com/privacy-policy">privacy policy</a> before you install the add-on.</p><p>Happy merging!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9dad860cd7ce" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>