<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Neil Avery on Medium]]></title>
        <description><![CDATA[Stories by Neil Avery on Medium]]></description>
        <link>https://medium.com/@neil.avery_99?source=rss-68d2da6ed2c0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*RrtIwZX0ubr6tuNKlRB2SQ.jpeg</url>
            <title>Stories by Neil Avery on Medium</title>
            <link>https://medium.com/@neil.avery_99?source=rss-68d2da6ed2c0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 01:43:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@neil.avery_99/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[On the alignment of Data Mesh and the Hexagonal Architecture — using Kafka]]></title>
            <link>https://medium.com/@neil.avery_99/on-the-alignment-of-data-mesh-and-the-hexagonal-architecture-using-kafka-03716245eb35?source=rss-68d2da6ed2c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/03716245eb35</guid>
            <category><![CDATA[event-driven-architecture]]></category>
            <category><![CDATA[asyncapi]]></category>
            <category><![CDATA[data-mesh]]></category>
            <category><![CDATA[hexagonal-architecture]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Neil Avery]]></dc:creator>
            <pubDate>Sat, 03 Feb 2024 09:49:00 GMT</pubDate>
            <atom:updated>2024-02-14T14:50:57.124Z</atom:updated>
            <content:encoded><![CDATA[<h3>On the alignment of Data Mesh and the Hexagonal Architecture — using Kafka</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/1*qhAgt5MjiSpltpQwxR0mCg.jpeg" /><figcaption>Data Mesh and Hexagonal architecture alignment — it’s the full stack</figcaption></figure><p>In the dynamic landscape of data management and software architecture, two powerful paradigms have emerged as guiding principles for modern organizations — Data Mesh and <a href="https://en.wikipedia.org/wiki/Hexagonal_architecture_(software)">Hexagonal Architecture</a>. These methodologies, aligned closely with Domain-Driven Design (DDD), advocate for decentralized and modular structures within an organization.</p><p>This article explores the convergence of Data Mesh and <a href="https://en.wikipedia.org/wiki/Hexagonal_architecture_(software)">Hexagonal Architecture</a>, augmented by the strengths of Kafka and <a href="https://specmesh.io/">SpecMesh</a>, offering a comprehensive approach for building robust and resilient distributed systems.</p><h3>Data Mesh and Hexagonal Architecture Overview</h3><p>Before we dive into the synergies, let’s briefly recap the fundamentals of Data Mesh and Hexagonal Architecture.</p><ul><li><em>Data Mesh:</em> Rooted in DDD, Data Mesh promotes modeling organizations into sub-domains where data products are discovered and shared. It emphasizes decentralized data management and the autonomy of self-contained domains.</li></ul><p><a href="https://martinfowler.com/articles/data-mesh-principles.html">Data Mesh Principles and Logical Architecture</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*b72Hi2mH4CQRQ-Mjz3ElCA.png" /><figcaption>The Data Mesh</figcaption></figure><p><a href="https://martinfowler.com/articles/data-mesh-principles.html">Learn more about Data Mesh Principles and Logical Architecture</a></p><p><em>Hexagonal Architecture:</em> Also aligned with DDD, Hexagonal Architecture encapsulates subdomains of business logic into a single repository, function, or bounded context. This functional unit of the organization enhances adaptability and prepares systems for changes.</p><p>This is a great introduction: <a href="https://thecodest.co/blog/t-strong-he-power-of-hexagonal-architecture/">https://thecodest.co/blog/t-strong-he-power-of-hexagonal-architecture/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cJWX9uppXx38EPF7ugmr_A.png" /><figcaption>Netflix and the Hexagonal architecture</figcaption></figure><p><a href="https://netflixtechblog.com/ready-for-changes-with-hexagonal-architecture-b315ec967749">Ready for changes with Hexagonal Architecture</a></p><h3>Alignments Between Data Mesh and Hexagonal Architecture</h3><p>The convergence of Data Mesh and Hexagonal Architecture introduces key alignments, strengthening the principles of both paradigms. These include:</p><ul><li>Decentralization — Data mesh encourages decentralized data management and governance by domain or product teams. Similarly, hexagonal architecture partitions monoliths into autonomous, decentralized domains and bounded contexts.</li><li>Domain-focused — Data mesh promotes the concept of “data products” provided by self-contained domains that own their data and logic. This aligns with how hexagonal architecture compartmentalizes business capabilities into separate domains.</li><li>Loose coupling — Both paradigms emphasize loose coupling and well-defined interfaces between domains/bounded contexts. The ports and adapters in hexagonal architecture enable this loose coupling.</li><li>API-driven — Data mesh expects domains to provide APIs for their data products that other teams can consume. These APIs essentially act as the ports in a hexagonal architecture.</li><li>Technology heterogeneity — Data mesh allows different data storage technologies to be used where appropriate. Similarly, hexagonal architecture’s adapters permit heterogeneous implementations of things like persistence — except its better to integrate using Kafka (not ports or adapters).</li><li>Independently deployable — Domains in both paradigms can be developed and deployed independently without centralized coordination. The encapsulation enables this.</li><li>Testing — The hexagonal architecture lends itself well to testing domains in isolation. Data mesh also derives benefits from clear testing seams between domains.</li></ul><p>Both paradigms align on principles like decentralization, loose coupling, APIs, and localized ownership that lend themselves well to combining the two approaches in an organization.</p><h3>Kafka Strengthens the Bond</h3><p>Using Kafka for the data mesh implementation further aligns it with hexagonal architecture patterns. The streaming data mesh is the right approach for leveraging the key benefits of Kafka (i.e. central nervous system) — but also event-driven microservices and data pipelines. This means we use Kafka topics/streams for the Ports and Adapters of the Hex architecture.</p><ul><li>Kafka becomes the communication backbone between bounded contexts and domains.</li><li>The Kafka topics and contracts can define the interfaces between domains, acting as the ports in hexagonal architecture.</li><li>The Kafka producers and consumers become the adapters that translate between the domain’s internal model and the external Kafka representation.</li><li>Kafka Streams can be used to implement domain logic and data pipelines behind the interfaces while keeping the domains decoupled.</li><li>Schema Registry provides a canonical schema representation for data shared between domains, enabling governance and evolution.</li><li>Kafka enables loose coupling, asynchronous event-driven communication between domains as envisioned in both paradigms.</li><li>Kafka replication provides redundancy for the data mesh implementation to prevent single points of failure.</li><li>With Kafka, new data producers and consumers can be added non-invasively to the data mesh, aligning with the extensibility goals.</li><li>Kafka topics and contracts make it easier to reason about and test domain interfaces in isolation.</li></ul><p>Kafka is a natural technical enabler for both data mesh and hexagonal architecture principles, providing the loose coupling, extensibility, and robustness needed to implement these paradigms effectively. The synergies make Kafka a great choice to support these architectures.</p><h3>SpecMesh as an enabler for Data Mesh</h3><p><a href="https://specmesh.io/">SpecMesh</a> is an approach that builds upon the concepts of data mesh and applies them to APIs and services — specifically — the AsyncAPI. Here is how SpecMesh aligns with hexagonal architecture patterns:</p><ul><li>SpecMesh promotes creating “spec products” which are self-contained API and service specifications owned by individual teams (sub-domains). This aligns with hexagonal architecture’s autonomous domains and bounded contexts.</li><li>The specs define data products (_public topics) for accessing capabilities that align with the ports in a hexagonal architecture. <a href="https://github.com/specmesh/docs/wiki">https://github.com/specmesh/docs/wiki</a></li><li>Behind the interfaces, teams can implement services using optimal technologies — similar to hexagonal adapters. Using _private topics.</li><li>Specs are published in a catalog for discovery — aligning with registering adapter endpoints.</li><li>SpecMesh the AsyncAPI to document and version spec interfaces. This enables loose coupling between consumers and producers. It is enabled using GitOps.</li><li>Kafka can be used as the messaging backbone to connect specs and domains in a decoupled event-driven way.</li><li>Testing and simulation of specs are emphasized, which aligns with hexagonal architecture’s focus on testability.</li><li>Teams own end-to-end responsibility for their specs — including documentation, testing, versioning etc. This decentralized ownership aligns with both paradigms.</li></ul><p>SpecMesh promotes autonomy, encapsulation, and loose coupling just like hexagonal architecture — but targeted to APIs and Kafka. Combining both patterns allows building robust and resilient distributed systems.</p><h3>Independently Deployable Domains</h3><p>The encapsulation and clear boundaries promoted by these paradigms allow domains to be developed, tested, and deployed independently without centralized coordination. This simplifies and parallelizes development across large organizations.</p><p>Kafka helps by decoupling the domains from direct dependencies. Backwards compatible interface changes allow teams to ship updates frequently and independently.</p><h3>Improved Testability</h3><p>Hexagonal architecture enables testing domains in isolation using test doubles and stubs for external interfaces. Data mesh also benefits from the clear separation of concerns between domains.</p><p>SpecMesh emphasizes testability of specs through simulation using tools like SwaggerHub. Kafka environments can be simulated using TestContainers for integration testing. I&#39;ll be talking more about this in a later article.</p><p>This makes it easier to implement test automation at the domain level for rapid feedback as changes are made. Teams can focus testing on their domains.</p><h3>Comments from the ‘Data Mesh learning group — slack group’</h3><blockquote>&gt; <em>In “Data Mesh”, Zhamak says that a data product is an “architectural quantum”, i.e. a unit of ownership and deployment, and includes things like ETL jobs, as well as the tables they populate. Ideally you can use a platform for the “undifferentiated heavy lifting”. — Mike Williams</em></blockquote><p>This raises a really important point — whilst Data Mesh solutions normally miss the analytic/processing/business logic part — the Hexagonal unit — is exactly this. Its a deployable/functional unit that can be black-box tested, observed and publishes its own data products works well at scale and strikes the right balance between mono-lith and microlith.</p><p>But what is it called — this conflation of Data Production (or collection of) — and a hex-processing module. In microservice terms it might be called a ‘bounded context’ — however — personally speaking — this name is terrible. Whilst, ‘Aggregate’ might not be as ‘correct’ (see DDD sources) — it seems like a conceptual alignment in the general sense of the word. Asking ChatGPT — (🤦‍♂ ) — we get:</p><blockquote>“Aggregate” is a suitable term that could describe your deployable and functional unit. In the context you described, an “Aggregate” could refer to a component responsible for aggregating, processing, and publishing data products while maintaining modularity and scalability.</blockquote><blockquote>An Aggregate typically represents a cohesive unit of functionality that encapsulates data processing logic and produces meaningful data outputs. It aligns with the idea of a black-box unit that can be independently tested, observed, and scaled.</blockquote><h3>The Power of Convergence</h3><p>The harmonious integration of Data Mesh, Hexagonal Architecture, Kafka, and SpecMesh presents a powerful toolbox for designing large-scale, evolvable, decentralized, and resilient distributed systems. The synergies derived from combining these paradigms offer organizations a strategic advantage in navigating the complexities of constant change. As the technological landscape continues to evolve, this combination provides a forward-thinking strategy for data and software architecture, simplifying complexity and accelerating delivery.</p><p>Together, these paradigms reinforced with Kafka and SpecMesh allow the building of evolvable, decentralized, and resilient distributed systems at the enterprise scale. Adopting this combination helps simplify complexity and accelerate delivery as organizations adapt to constant change.</p><p>In summary, the synergies between data mesh, hexagonal architecture, SpecMesh, Kafka and AsyncAPI provide a powerful toolbox for robust large-scale platform design and domain-driven development.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=03716245eb35" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Datazone vs SpecMesh OS for Kafka (fight!)]]></title>
            <link>https://medium.com/@neil.avery_99/aws-datazone-vs-specmesh-os-for-kafka-fight-7fab4b140f28?source=rss-68d2da6ed2c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/7fab4b140f28</guid>
            <category><![CDATA[streaming]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[data-mesh-architecture]]></category>
            <category><![CDATA[data-mesh]]></category>
            <category><![CDATA[event-driven-architecture]]></category>
            <dc:creator><![CDATA[Neil Avery]]></dc:creator>
            <pubDate>Wed, 11 Oct 2023 14:42:43 GMT</pubDate>
            <atom:updated>2023-10-11T14:42:43.897Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XW8Ti8GQ1gBiEZ4vbSbKUQ.jpeg" /><figcaption>Data Mesh in the Cloud using AWS DataZone or SpecMesh for Apache Kafka</figcaption></figure><p>AWS recently released “<a href="https://docs.aws.amazon.com/datazone/latest/userguide/what-is-datazone.html">AWS DataZone</a>”, something, that much like <a href="https://specmesh.io/">SpecMesh</a> enables the sharing of resources, self-governance, and discoverability — and sounds like what the industry calls a ‘Data Mesh’. That’s because this is exactly what it is.</p><blockquote>“Data architectures were mainly designed around technologies rather than business domains in the past. This changed in 2019 when Zhamak Dehghani introduced the <a href="https://aws.amazon.com/what-is/data-mesh/">data mesh</a>. Data mesh is an application of the Domain-Driven-Design (DDD) principles to data architectures: Data is organized into data domains and the data is the product that the team owns and offers for consumption.</blockquote><blockquote>A data mesh architecture unites disparate data sources within an organization through centrally managed data-sharing and governance guidelines. Business functions can maintain control over how shared data is accessed because data mesh also solves advanced data security challenges through distributed, decentralized ownership.”</blockquote><p>Source: <a href="https://aws.amazon.com/blogs/architecture/lets-architect-architecting-a-data-mesh/">https://aws.amazon.com/blogs/architecture/lets-architect-architecting-a-data-mesh/</a></p><blockquote>“Data mesh is a pattern for defining how organizations can organize around data domains with a focus on delivering data as a product. However, it may not be the right pattern for every customer.”</blockquote><p>Source: <a href="https://martinfowler.com/articles/data-mesh-principles.html">https://martinfowler.com/articles/data-mesh-principles.html</a></p><p>In my opinion, the significance of AWS DataZone extends beyond its technological aspects. It serves as a validation of the idea that Data Mesh should be “productised”. Numerous vendors assert that creating a Data Mesh necessitates extensive, intricate business processes, human resources, and various other elements. However, within the <a href="https://specmesh.io/">SpecMesh</a> initiative, I firmly believe that a substantial portion of Data Mesh implementation can and should be streamlined through <a href="https://about.gitlab.com/topics/gitops/">automation</a> (GitOps), <a href="https://www.asyncapi.com/">specifications</a>, and the use of simple, yet specialized tools (like SpecMesh). This progression aligns with the natural evolution of technology and its adoption as a standard solution — all part of the journey to commoditization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nbE9uqi0MIZ3loHC" /></figure><p>Source: <a href="https://d2908q01vomqb2.cloudfront.net/fc074d501302eb2b93e2554793fcaf50b3bf7291/2023/03/07/Screen-Shot-2023-03-07-at-11.56.22-AM.png">https://d2908q01vomqb2.cloudfront.net/fc074d501302eb2b93e2554793fcaf50b3bf7291/2023/03/07/Screen-Shot-2023-03-07-at-11.56.22-AM.png</a></p><h3>Let’s compare and contrast AWS Datazone against SpecMesh.io</h3><p><strong>The AWS pitch:</strong></p><blockquote>“Amazon DataZone is a data management service that makes it faster and easier for you to catalog, discover, share, and govern data stored across AWS, on-premises, and third-party sources. With Amazon DataZone, administrators and data stewards who oversee the organisation’s data assets can manage and govern access to data using fine-grained controls. These controls help ensure access with the right level of privileges and context. Amazon DataZone makes it easy for engineers, data scientists, product managers, analysts, and business users to share and access data throughout an organization so they can discover, use, and collaborate to derive data-driven insights.”</blockquote><p>Source: <a href="https://docs.aws.amazon.com/datazone/latest/userguide/what-is-datazone.html">https://docs.aws.amazon.com/datazone/latest/userguide/what-is-datazone.html</a></p><p><strong>The SpecMesh pitch:</strong></p><blockquote>SpecMesh is an open-source tool based on the AsyncAPI standard, designed to facilitate the creation of a “streaming data mesh.” It harnesses GitOps workflows to enable the provisioning of specifications (referred to as Data Products) within Apache Kafka. These specifications essentially embody a Data Product, which comprises organized and interconnected topics (referred to as Assets) and facilitates the dissemination of metadata to a data catalogue, promoting sharing and discovery.</blockquote><p>Both achieve similar goals — build a Data Mesh using a standard set of tools and principles. AWS DataZone does it the “AWS way” and <a href="https://specmesh.io/">SpecMesh</a> does it the OpenSource, <a href="https://www.asyncapi.com/">specification-based</a>, GitOps way.</p><p>Let’s delve into the shared Data Mesh characteristics found in both technologies. These characteristics, often referred to as <a href="https://www.starburst.io/info/distributed-data-mesh-resource-center">“traits” or “four pillars”</a> represent a consensus within the industry. They include Domain Ownership, Data Product, Self-Governance, and Discovery.</p><h3>Domain alignment (Domain ownership)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AGgO6thspkvLzrSv" /></figure><p>Source: <a href="https://www.datamesh-architecture.com/images/datamesharchitecture.png.webp">https://www.datamesh-architecture.com/images/datamesharchitecture.png.webp</a></p><p><strong>Definition: </strong>As seen above — the ‘domain’ is front and center in the concept of Data Mesh. It is the collection of resources owned by a functional business area, a domain unit</p><blockquote>“Data mesh follows the seams of organizational units as the axis of decomposition. Our organizations today are decomposed based on their business domains. “<br>From: <a href="https://martinfowler.com/articles/data-mesh-principles.html">https://martinfowler.com/articles/data-mesh-principles.html</a></blockquote><p>My Perspective — the concept of domain orientation, and while I might be a bit old-school, <a href="https://www.eventstorming.com/">Event Storming</a> and resultant, <a href="https://en.wikipedia.org/wiki/Domain-driven_design">Domain-Driven Design</a> (DDD) — are the core elements of any large-scale system architecture. We have been building systems like this for years, the Data Mesh spin places a distinct focus on the concept of ‘data’</p><p>See: <a href="https://en.wikipedia.org/wiki/Domain-driven_design">https://en.wikipedia.org/wiki/Domain-driven_design</a></p><blockquote><strong>Aws DataZone: </strong>“An Amazon DataZone <strong>domain</strong> is the organizing entity for connecting together your assets, users, and their projects. With Amazon DataZone domains, you have the flexibility to reflect the data and analytics needs of your organizational structure, whether it’s creating a single Amazon DataZone domain for your enterprise or multiple datazone; domains for different business units or teams.”</blockquote><blockquote><strong>SpecMesh: </strong>In SpecMesh, a domain is encapsulated within an AsyncAPI specification. This specification is a compilation of streaming resources (referred to as topics/assets) that are associated with a specific functional unit, such as “Procurement” or “Onboarding.” Each of these functional units is comprised of interrelated services and data components. In the context of AsyncAPI, these ‘assets’ correspond to Kafka topics along with their respective attributes and associated schemas.</blockquote><p>An example specmesh.yml file:</p><pre>asyncapi: &#39;2.5.0&#39;<br>id: &#39;urn:acme.simple_range.life_enhancer&#39;<br>channels:<br>  _public/user_signed_up:<br>    bindings:<br>      kafka:<br>        partitions: 3<br>        replicas: 1<br>        configs:<br>          retention.ms: 999000<br>    publish:<br>      summary: Inform about signup<br>      operationId: onSignup<br>      message:<br>       payload:<br>        $ref: &quot;/schema/acme.simple_range.life_enhancer._public.user_signed_up.avsc&quot;</pre><p>Source: <a href="https://github.com/specmesh/docs/wiki">https://github.com/specmesh/docs/wiki</a></p><p>Provision using <a href="https://specmesh.io/">SpecMesh</a> CLI to create</p><ul><li>Topic: acme.simple_range.life_enhancer._public.user_signed_up</li><li>Schema: /schema/acme.simple_range.life_enhancer._public.user_signed_up.avsc</li><li>Principle: “acme.simple_range.life_enhancer”</li><li>Grant public READ_ONLY (<a href="https://docs.confluent.io/platform/current/kafka/authorization.html">ACL</a>) access to this topic (by any other principle)</li></ul><p>Some more examples of domains:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*51fLliVdOec1dpoZ" /></figure><p>Source: <a href="https://martinfowler.com/articles/data-mesh-principles/domains.png">https://martinfowler.com/articles/data-mesh-principles/domains.png</a></p><h3><strong>Data as a product (data product)</strong></h3><p><strong>Definition: </strong>Each domain owns its data end to end. They’re responsible for building, operating, serving, and resolving any issues arising from the use of their data. Data accuracy and accountability lie with the data owner within the domain.</p><blockquote>According <a href="https://www.manning.com/books/data-mesh-in-action">to J. Majchrzak et al.</a>, a <em>“data product is an autonomous, read-optimized, standardized data unit containing at least one dataset (Domain Dataset), created for satisfying user needs”.</em></blockquote><p><em>Source: </em><a href="https://www.manning.com/books/data-mesh-in-action"><em>https://www.manning.com/books/data-mesh-in-action</em></a></p><p>As an example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*MRwL33HA02KjGN9z" /></figure><p>Source: <a href="https://martinfowler.com/articles/data-mesh-principles/data-product-components.png">https://martinfowler.com/articles/data-mesh-principles/data-product-components.png</a></p><p>As a point of background information, there are several efforts to define a data product and related functions, using heavily borrowed semantics from the OpenAPI and <a href="https://www.asyncapi.com/">AsyncAPI</a> specs — this project is interesting:</p><ul><li><a href="https://datacontract.com/">https://datacontract.com/</a></li><li><a href="https://www.acryldata.io/blog/data-contracts-in-datahub-combining-verifiability-with-holistic-data-management?utm_source=datahub&amp;utm_medium=referral&amp;utm_content=blog">https://www.acryldata.io/blog/data-contracts-in-datahub-combining-verifiability-with-holistic-data-management?utm_source=datahub&amp;utm_medium=referral&amp;utm_content=blog</a></li></ul><p>Acryl Datas view on Data Products: (creators of <a href="https://datahubproject.io/">DataHub</a>)</p><blockquote>“Data Products within DataHub help you “<strong>define a boundary around these assets to create and define your own Data Product</strong>”</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*94R1WSyGWvud_Pll" /></figure><p>Source: <a href="https://www.acryldata.io/blog/data-products-in-datahub-everything-you-need-to-know">https://www.acryldata.io/blog/data-products-in-datahub-everything-you-need-to-know</a></p><p>Examples of data products include:</p><ul><li>Personalized recommendations on an e-commerce website based on a customer’s purchase history and browsing behavior (<a href="https://app.snowflake.com/marketplace/listing/GZT0ZRRV7TT/sharethis-ecommerce-shopping-behavior-interest-data-trial">example</a>)</li><li>Real-time traffic updates based on GPS data from millions of connected vehicles. (<a href="https://datarade.ai/data-categories/vehicle-location-data">example</a>)</li><li>Predictive maintenance systems that use data from sensors to identify potential equipment failures before they occur (<a href="https://marine-digital.com/predictive_maintenance_of_equipment">example</a>)</li><li>Healthcare analytics platforms that use data from electronic health records to improve patient outcomes. (<a href="https://www.definitivehc.com/data-products">example</a>)</li></ul><p><strong>Aws DataZone <br></strong>Datazone doesn’t really define a “Data product” as such — instead, it seeks to declare ‘assets’ as part of the ‘domain’, for example, an S3 bucket — to which ‘consumers’ can discover and request access (IAM) to — from within the DataZone portal. The portal owner, then grants access to that account (and also later revoke it).</p><p>See:</p><ul><li><a href="https://docs.aws.amazon.com/datazone/latest/userguide/create-data-asset-manually.html">https://docs.aws.amazon.com/datazone/latest/userguide/create-data-asset-manually.html</a></li><li><a href="https://docs.aws.amazon.com/datazone/latest/userguide/create-asset-types.html">https://docs.aws.amazon.com/datazone/latest/userguide/create-asset-types.html</a></li></ul><p><strong>SpecMesh<br></strong><a href="https://specmesh.io/">SpecMesh</a> takes a distinct approach by simplifying the asset types to encompass only one entity: the <a href="https://www.conduktor.io/kafka/kafka-topics/">Kafka topic</a>. These topics are defined within the SpecMesh <a href="https://www.asyncapi.com/">AsyncAPI</a>.yml file, each adhering to a templated and enforced topic structure. For instance, ‘com.hr.onboarding._public.user’ exemplifies this structure, where the domain owner (‘hr.onboarding’) is transparently conveyed through the topic name. Furthermore, the ‘_public’ designation signifies public data, and the data type, in this case, is ‘user’ events, specified by the schema.</p><p>Consequently, we can infer that in SpecMesh, the ‘data product’ comprises the collection of ‘_public’ and ‘_protected’ topics within a domain owner’s specification. These specifications are made available in a data catalog for the purpose of discovery, including the associated schemas.</p><p>A ‘data product’ can also be defined to encompass exclusively the ‘_protected’ and ‘_public’ topics, granting access to designated domain users, such as other specification owners. This approach fosters self-governance, a topic I will delve into shortly.</p><p>Arguably, the most vital point to consider is that the ‘Spec’ serves as a data model representing the amalgamation of resources, including topics, their hierarchy, permissions, and data formats (schemas). All these ‘assets’ or resources support an extensive metadata model aimed at capturing business terminology and a glossary. When published to a data catalog, this metadata enriches communication and facilitates the sharing of significant attributes and semantics across the organization. Further elaboration on this aspect follows.</p><h3>(Self) Data Governance</h3><p><strong>Definition: </strong>Data governance ensures data is secure, accurate, and not misused. The technical implementation of data governance such as collecting lineage, validating data quality, encrypting data at rest and in transit, and enforcing appropriate access controls can be managed by each of the data domains. However, central data discovery, reporting, and auditing is needed to make it simple for users to find data and for auditors to verify compliance.</p><blockquote><strong>AWS DataZone<br></strong>“The central data governance account stores a data catalogue of all enterprise data across accounts, and provides features allowing producers to <a href="https://docs.aws.amazon.com/lake-formation/latest/dg/register-location.html">register</a> and <a href="https://docs.aws.amazon.com/lake-formation/latest/dg/creating-database.html">create</a> catalogue entries with AWS Glue from all their S3 buckets. No data (except logs) exists in this account. Lake Formation centrally defines security, governance, and auditing policies in one place, enforces those policies for consumers across analytics applications, and only provides authorization and session token access for data sources to the role that is requesting access. Lake Formation also provides uniform access control for enterprise-wide data sharing through <a href="https://docs.aws.amazon.com/lake-formation/latest/dg/regranting-shared-resources.html">resource shares</a> with centralized governance and auditing.”</blockquote><p><strong>SpecMesh<br></strong>The SpecMesh approach for sharing is to conflate permissions (ACLs) into the topic structure (or attach to metadata ‘tags’ <a href="https://www.asyncapi.com/docs/reference/specification/v3.0.0-next-major-spec.14#tagsObject">https://www.asyncapi.com/docs/reference/specification/v3.0.0-next-major-spec.14#tagsObject</a> ). The model follows well-established file system/programmer semantics, of _private, _protected, and _public, Private and public operate as expected, _public supports READ_ONLY across the organisation, and _private READ/WRITE by the `domain owner`. This operation is enforced by the ‘principle’ — the principle is the ‘id’ field within the spec, and is also used to prefix topic names.</p><p>For example, topic:</p><pre>com.hr.onboarding._public.user</pre><p>is structured as:</p><pre>{domain-id/principle}{permission}{data/event-type}</pre><p>When ‘protected’ access is required, then the requestor needs to locate the domain owner via the data-catalogue and start the relevant access-request workflow. Ultimately, leading to the domain owner adding a ‘grant-access’:{domain-id} to their spec.</p><h3>Discoverability and sharing</h3><p><strong>Definition:</strong> The data catalogue contains the datasets and assets registered by data domain producers, including supporting metadata such as lineage, data quality metrics, ownership information, and business context. It is through Data catalogues that we enable discovery, governance, and collaboration across an organisation.</p><blockquote><strong>AWS DataZone</strong><br>“In order to use Amazon DataZone to catalog your data, you must first bring your data (assets) as inventory of your project in Amazon DataZone. Creating inventory for a particular project, makes the assets discoverable only to that project’s members. Project inventory assets are not available to all domain users in search/browse unless explicitly published. After creating a project inventory, data owners can curate their inventory assets with the required business metadata by adding or updating business names (asset and schema), descriptions (asset and schema), read me, glossary terms (asset and schema), and metadata forms.</blockquote><blockquote>The next step of using Amazon DataZone to catalog your data, is to make your project’s inventory assets discoverable by the domain users. You can do this by publishing the inventory assets to the Amazon DataZone catalog”</blockquote><p>Source: <a href="https://docs.aws.amazon.com/datazone/latest/userguide/publishing-data.html">https://docs.aws.amazon.com/datazone/latest/userguide/publishing-data.html</a></p><p><strong>SpecMesh<br></strong>To make Data Products (specs) discoverable, SpecMesh specs need to be published to a Data Catalogue of your choice. Two of the most popular catalogues are LinkedIn DataHub, and also AWS Glue. In both cases, the spec is published, meta-data extracted and annotated, as well as schemas and associated metadata extracted and published.</p><p>See:</p><ul><li><a href="https://aws.amazon.com/glue/">https://aws.amazon.com/glue/</a></li><li><a href="https://datahubproject.io/">https://datahubproject.io/</a></li></ul><h3>What’s the difference then?</h3><p><a href="https://docs.aws.amazon.com/datazone/latest/userguide/what-is-datazone.html">AWS DataZone</a> is an AWS native data asset publishing, and sharing platform for enterprise-wide collaboration. It supports all the expected functionality required of a Data Mesh — and it does it well, in the AWS way. This means, IAM, Roles, and portals. If there is an API to control and automate it would be a much sweeter offering, because it enables us to enter the world of GitOps. I’m not digging deep enough — but I need APIs, and docker containers so I can build and test this stuff in my CI/CD.</p><p>Below is a lovely AWS “Market-itecture” diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_jYEK0PSaqq-wiOl" /></figure><p>Source: <a href="https://d2908q01vomqb2.cloudfront.net/b6692ea5df920cad691c20319a6fffd7a4a766b8/2021/07/08/bdb1297-image001.png">https://d2908q01vomqb2.cloudfront.net/b6692ea5df920cad691c20319a6fffd7a4a766b8/2021/07/08/bdb1297-image001.png</a></p><p><a href="https://specmesh.io/"><strong>SpecMesh</strong></a>, conversely, is <a href="https://www.asyncapi.com/">spec-driven</a>, cloud-agnostic, GitOps-centric, enforces Apache Kafka resource best practices (structured topics) and is Kafka only — it is homogenous. This simplification closely aligns with industry-adopted best practices (GitOps, Specs), local development (testing, CI/CD), and architecture principles (domain orientation). The spec provides a data model that conflates sharing as well as meta-data which models the collection of topics (assets) as a data product. It is concise and uncomplicated.</p><p>Below is a SpecMesh “Market-itecture” diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JzXoj6gTNTBAXn2A" /></figure><p>Source: <a href="https://specmesh.io/images/mesh/ov-enterprise.png">https://specmesh.io/images/mesh/ov-enterprise.png</a></p><h3>In summary</h3><p>Both are good — you choose /j</p><p>Realistically if you are an AWS cloud-native — then AWS DataZone probably makes sense. However, if you are “events as a backbone”, homogenised data sources, and event-driven based upon Apache Kafka (or <a href="https://aws.amazon.com/msk/">MSK</a>) then take a good look at SpecMesh. Ideally, head on over to the <a href="https://github.com/specmesh">GitHub</a> page, “star” the repo, and take it for a test drive with the <a href="https://github.com/specmesh/getting-started-apachekafka">Apache QuickStart</a>.</p><p><a href="https://github.com/specmesh/specmesh-build">https://github.com/specmesh/specmesh-build</a></p><p><em>We are just getting started!</em></p><p>Interesting Data Mesh resources:</p><ul><li><a href="https://aws.amazon.com/blogs/big-data/how-godaddy-built-a-data-mesh-to-decentralize-data-ownership-using-aws-lake-formation/">Go Daddy DataMesh</a></li><li><a href="https://aws.amazon.com/blogs/architecture/lets-architect-architecting-a-data-mesh/">AWS — Lets architect a Data Mesh</a></li><li><a href="https://blog.datahubproject.io/enabling-data-discovery-in-a-data-mesh-the-saxo-journey-451b06969c8f">Saxo Bank — Data Discovery — the Saxo Journey</a></li><li><a href="https://www.confluent.io/blog/distributed-domain-driven-architecture-data-mesh-best-practices/">DDD and Data Mesh Best Practices</a></li><li><a href="https://aws.amazon.com/datazone/features/data-access/">AWS DataZone and Data access</a></li><li><a href="https://martinfowler.com/articles/data-mesh-principles.html">Data Mesh Principles and Logical Architecture</a>, Zhamak Dehghani</li><li><a href="https://www.jamesserra.com/archive/2021/02/data-mesh/">Data Mesh Defined</a>, James Serra</li><li><a href="https://www.weave.works/technologies/gitops/">Guide to GitOps</a>, WeaveWorks</li><li><a href="https://www.domainlanguage.com/">Domain Language</a>, domainlanguage.com</li><li><a href="https://www.thoughtworks.com/radar/techniques/data-integrity-at-the-origin">Data Integrity at Origin</a>, ThoughtWorks Technology Radar</li><li><a href="https://github.com/linkedin/datahub/">DataHub Repo</a>, LinkedIn</li><li><a href="https://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html">Schema Evolution in Avro, Protocol Buffers and Thrift</a>, Martin Kleppman</li><li><a href="https://martinfowler.com/bliki/TwoHardThings.html">TwoHardThings</a>, Martin Fowler</li><li><a href="https://spec.edmcouncil.org/fibo/">Financial Industry Business Ontology</a>, EDM Council</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7fab4b140f28" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Patterns For Kafka-centric, Event-driven Microservice Architectures]]></title>
            <link>https://medium.com/@neil.avery_99/patterns-for-kafka-centric-event-driven-microservice-architectures-c5dbd6b79182?source=rss-68d2da6ed2c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/c5dbd6b79182</guid>
            <category><![CDATA[kafka-streams]]></category>
            <category><![CDATA[data-mesh]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[domain-driven-design]]></category>
            <category><![CDATA[event-driven-architecture]]></category>
            <dc:creator><![CDATA[Neil Avery]]></dc:creator>
            <pubDate>Tue, 29 Aug 2023 02:21:25 GMT</pubDate>
            <atom:updated>2023-08-29T02:38:21.591Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*McMrUYzrtMwye7BTJQTpjw.jpeg" /><figcaption>Organise company architecture to reflect the domain functionality using DDD</figcaption></figure><p>Here is a compilation of patterns and valuable approaches I have developed through extensive experience on numerous Kafka projects. More information can be found on the blogs at <a href="https://specmesh.io/">specmesh.io</a>.</p><p>Feel free to contact me for further insights.</p><p><strong>Big Picture Patterns</strong></p><ul><li>Domain orientation — apply a top-to-bottom stack concept that puts all the related things in a domain model of the same <strong>hierarchical domain label</strong> (i.e. org.acme.accounts):<br>-<strong> Service clusters: </strong>EKS, service names, operational events (logs -&gt; CloudEvents, etc.)<br>- <strong>Security </strong>— service roles, perms, ACLs<br>- <strong>Provisioning:</strong> use GitOps style provisioning of data-infra that models the domain — see <a href="https://specmesh.io/">specmesh.io</a><br>- <strong>Data infra:</strong> topic structure, schema subject — see <a href="https://specmesh.io/">specmesh.io</a><br>- B<strong>uild infra:</strong> build pipeline, ‘domain-level’ tests, black-box testing, acceptance testing<br>- <strong>Development infra:</strong> source repo i.e. the collection of related services</li></ul><p><strong>Operational Patterns</strong></p><ul><li><strong>DQL — Dead Letter Queue:</strong> Failing consumers place those events onto a special queue. (org.acme.accounts.Booking → org.acme.accounts.Booking_DLQ)</li><li><strong>Retry Queue:</strong> derives from DLQ — but identifies events that can be retried i.e. dependencies now resolve or a processing ‘fix’ has been applied. org.acme.accounts.Booking_DQL → org.acme.accounts.Booking_RETRY</li></ul><p><strong>Orthogonal Planes</strong></p><ul><li><strong>Notification plane:</strong> Send business events and alerts — also ties into run-books, directed to a specific audience (front-office), captures exceptions, and links to associated docs at the time of the event. Fits with tooling <a href="https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-user-guide.html">CloudTrail</a>. Use a dedicated topic with post-fix xxx_NOT</li><li><strong>Instrumentation plane:</strong> Send metrics, and timestamps to related topics<br>Use a dedicated topic with post-fix xxx_INST</li><li><strong>Trace plane:</strong> timestamp+correlation-id+ topic-partition+ offset+node+service+id. Use a dedicated topic with post-fix xxx_TRC</li><li><strong>Control plane:</strong> stop, start, pause services. Use a dedicated topic with post-fix xxx_CTRL</li></ul><p><strong>Producer-Consumer, Microservice events</strong></p><ul><li><strong>Compacting producer:</strong> compacts the batch upon sending — reduces Cloud costs by reducing data of high volume data (i.e. FX market data feeds)</li><li><strong>Back pressure loop:</strong> allow producers to observe lag and determine if they should slow down or alert</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c5dbd6b79182" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SpecMesh — Provisioning Kafka resources using the AsyncAPI Spec]]></title>
            <link>https://medium.com/@neil.avery_99/specmesh-provisioning-kafka-resources-using-the-asyncapi-spec-4e432c5fe950?source=rss-68d2da6ed2c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/4e432c5fe950</guid>
            <category><![CDATA[event-driven-architecture]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[asyncapi]]></category>
            <category><![CDATA[data-mesh]]></category>
            <dc:creator><![CDATA[Neil Avery]]></dc:creator>
            <pubDate>Mon, 05 Jun 2023 13:17:41 GMT</pubDate>
            <atom:updated>2023-06-05T13:18:57.771Z</atom:updated>
            <content:encoded><![CDATA[<h3>SpecMesh: Provisioning Kafka resources using the AsyncAPI Spec</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BoVz8GW_7mBgrapmy-0psw.png" /><figcaption>SpecMesh provisions Kafka Resources using the AsyncAPI spec as the source of truth</figcaption></figure><p>In this blog, I will discuss how SpecMesh utilizes the <a href="https://www.asyncapi.com/">AsyncAPI</a> spec to provision Kafka resources and how it extends some of AsyncAPI spec functionality through its textual structuring introduced via regular text patterns. The AsyncApi spec is normally used for documentation purposes — we regularly see it being used for capturing existing deployments — SpecMesh inverts this to a position where it is used as the source of truth in a <a href="https://about.gitlab.com/topics/gitops/">GitOps</a> workflow and the provisioning of Kafka resources and governance (it can also export a spec).</p><p>In the image below is a snippet of a spec. One of the first elements to note is the ID field in the AsyncAPI spec, which is used recurrently for various purposes.</p><pre>asyncapi: &#39;2.5.0&#39;<br>id: &#39;urn:acme:simple_range:life_enhancer&#39;<br>channels:<br>  _public/user_signed_up:<br>    bindings:<br>      kafka:<br>        partitions: 3<br>        replicas: 1<br>        configs:<br>          retention.ms: 999000<br>    publish:<br>      summary: Inform about signup<br>      operationId: onSignup<br>      message:<br>        payload:<br>          $ref: &quot;/schema/acme.simple_range.life_enhancer._public.user_signed_up.avsc&quot;</pre><p>The primary purpose of this ID field extends beyond merely understanding the name or identification of the data product. It also aims to structurally map it within the organisation for domain ownership (and functionality). For instance, in our case, the data product includes <em>“ACME, Simple Range, and Life Enhancer”</em>. In this fictitious structure, it’s broken down as company.product-group.product. With DDD (Domain-Driven Design), we aim to decompose things into aggregates, which are groups of related functionality (a cluster of services ideally from the same aggregate level repository, thus supporting black box testing) rather than relying on organizational groupings.</p><p>The ID is prefixed to individual channels or topics, as they are known in Kafka. Each of the channels owned by the Life Enhancer product is prefixed with either a _public or _private keyword. Additionally, the _protected keyword is also employed for self-governance purposes where restricting access to specific ‘domain ids’ is required.</p><p>Within each section, we have a Kafka configuration section and the schema related to it. The schema captures the data format of the event set to be published. Hence, the ID serves multiple purposes, with the key purpose being the principle. This allows for clear ownership when any resource is provisioned on behalf of this data product. Topics are prefixed with the principal, schemas are published into the schema registry prefixed with a schema principal, and ACLs also use the prefix. ACLs are governed or controlled by the keywords public, private, or protected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bp4Hl2wxD-oRZu9baGrB7A.png" /><figcaption>Running the provision command</figcaption></figure><p>Suppose our developer has designed the data model for their data product. The next step is to provision an environment (local machine, or cluster). The <a href="https://github.com/specmesh/specmesh-build/blob/main/cli/README.md">provision</a> command, part of the SpecMesh CLI, utilizes the <a href="https://kafka.apache.org/documentation/#adminapi">Kafka admin client</a>. This admin client can be used against any Kafka infrastructure, such as MSK, Open Source Apache Kafka, Confluent Cloud, Red Panda, or others. The provision command, executed from the command line, through Docker, or a GitHub workflow as part of a pull request, then provisions the topics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JnPUdKBONYa4k7sMRk_Eng.png" /></figure><p>You will see the principal ID, ACME Simple Range Live Enhancer, prefixing each of the topics. The entire thing is concatenated together, and the configuration of retention and cleanup policies is specified. Schemas are then published into the Schema Registry, where ownership is clearly identified by the principal or the data product ID. ACLs, however, are slightly different. Controlled by public, private, and protected, the application owner or the data product owner in the case of protected can self-govern and use tags within the spec to grant access to other domains. When SpecMesh provisions this data product, it configures those ACLs accordingly (_private, _public, protected). Role-based Access (RBAC) is currently being developed and will be available soon.</p><p>Key points:</p><ol><li>AsyncAPI extension: id is used to identify the data-product using domain driven thinking, it also represents the security principle (secrets etc), as well as the Aggregate in the context of an business’s function (not an org map). This is one of the pillars of DataMesh</li><li>Ownership of ‘data product’ resources is established by prefixing the id to all resources</li><li>AsyncAPI extension: Prefixing channels (topics) using _private, _public, and _protected provides sharing controls and suppress the need for understanding ACL configuration.</li><li>AsyncAPI extension: _protected topics provide a means of fine-grained self-governance using the tag: grant-access pattern. This is one of the pillars of DataMesh</li></ol><p>Visit <a href="http://specmesh.io/">specmesh.io</a> for more details!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4e432c5fe950" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SpecMesh: Apache Kafka, GitOps, and AsyncAPI = A Powerful Combination]]></title>
            <link>https://medium.com/@neil.avery_99/specmesh-apache-kafka-gitops-and-asyncapi-a-powerful-combination-bf057f4033a3?source=rss-68d2da6ed2c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/bf057f4033a3</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[gitops]]></category>
            <dc:creator><![CDATA[Neil Avery]]></dc:creator>
            <pubDate>Mon, 15 May 2023 15:41:51 GMT</pubDate>
            <atom:updated>2023-05-15T15:41:51.142Z</atom:updated>
            <content:encoded><![CDATA[<p>Apache Kafka is a distributed streaming platform that has become a popular choice for building real-time data pipelines and streaming applications. GitOps is a software development methodology that emphasizes the use of Git for version control and the automation of deployment and management tasks. AsyncAPI is a specification for describing asynchronous APIs that can be used to design, document, and test message-driven systems. Together, these three technologies can provide a powerful combination for building scalable, reliable, and maintainable messaging systems. This is all great in theory, but the reality is that nothing out there really combines them together properly until now….</p><p><strong>Apache Kafka</strong><br>Apache Kafka is a distributed streaming platform that was originally developed at LinkedIn and later open-sourced by the Apache Software Foundation. Kafka is designed to handle large amounts of data in real-time by providing a distributed and fault-tolerant architecture. Kafka uses a publish-subscribe model for sending and receiving messages, and it can handle millions of messages per second with low latency.</p><p>Kafka has become a popular choice for building real-time data pipelines and streaming applications because of its scalability, reliability, and performance. Kafka is used by many companies for a variety of use cases, including log aggregation, stream processing, event sourcing, and messaging.</p><p><strong>GitOps</strong><br>GitOps is a software development methodology that emphasizes the use of Git for version control and the automation of deployment and management tasks. With GitOps, all changes to the system are made through Git, and the system is automatically updated based on changes to the Git repository. This approach provides several benefits, including version control, auditability, and repeatability.</p><p>GitOps can be used to manage the deployment and configuration of Kafka clusters, as well as the deployment of Kafka applications. With GitOps, changes to the Kafka cluster configuration or application code can be made through Git, and the changes can be automatically deployed to the cluster.</p><p><strong>AsyncAPI</strong><br>AsyncAPI is a specification for describing asynchronous APIs that can be used to design, document, and test message-driven systems. AsyncAPI provides a standard way to describe the messages that are exchanged between different components of a system, including topics, events, and message schemas.</p><p>AsyncAPI can be used to design and document Kafka topics and messages, as well as to generate client and server code for Kafka applications. AsyncAPI can also be used to test Kafka applications by generating test cases based on the AsyncAPI specification.</p><p><strong>The problem</strong></p><p>If you have been using Kafka for a while, you will understand the many challenges and complexities surrounding such a seemingly simple concept as a distributed log-append structure. Generally, I often observe that learning on the job can lead to the following mistakes:</p><ul><li>Poorly named topic structures (or a lack of structure or convention)</li><li>Absence of schemas</li><li>Versioning of topics</li><li>Lack of security measures</li><li>Unclear ownership (i.e., it’s not clear who owns what)</li></ul><p>And this is just the tip of the iceberg…</p><p>SpecMesh seeks to address all of these issues. Moreover, it incorporates some of the best features from GitOps and AsyncAPI to create a development toolkit and methodology for building your Kafka apps in the right way.</p><p><strong>SpecMesh</strong></p><p><a href="https://specmesh.io/">SpecMesh.io</a> leverages these technologies and practices, doing so allows it to simplify how your Kafka app is described (topics are structured), the app has a domain (DDD, Aggregate) owner, and permissions (ACLs) and is conflated into the topic structure.</p><p>An example spec</p><pre>asyncapi: &#39;2.5.0&#39;<br>id: &#39;urn:acme:lifestyle:onboarding&#39;<br>info:<br>  title: ACME Lifestyle Onboarding<br>  version: &#39;1.0.0&#39;<br>  description: |<br>    The ACME lifestyle onboarding app that allows stuff - see this url for more detail.. etc<br><br>channels:<br>  _public/user_signed_up:<br>    bindings:<br>      kafka:<br>        partitions: 3<br>        replicas: 1<br>        retention: 1<br>        configs:<br>          cleanup.policy: delete<br><br>    publish:<br>      message:<br>        payload:<br>          $ref: &quot;/schema/simple.schema_demo._public.user_signed_up.avsc&quot;<br><br>  _private/user_checkout:<br>    publish:<br>      message:<br>        payload:<br>          $ref: &quot;/schema/simple.schema_demo._public.user_checkout.yml&quot;<br><br><br>  _protected/purchased:<br>    publish:<br>      summary: Humans purchasing food - note - restricting access to other domain principles<br>      tags: [<br>        name: &quot;grant-access:.some.other.domain.root&quot;<br>      ]<br>      message:<br>        name: Food Item<br>        tags: [<br>          name: &quot;human&quot;,<br>          name: &quot;purchase&quot;<br>        ]</pre><p><strong>Conclusion</strong><br>Apache Kafka, GitOps, and AsyncAPI provide a powerful combination but until now nothing has really given developers a modeling expression language for structuring their Kafka apps while leveraging these tools correctly. SpecMesh strives to build on these and provide a platform upon which organizations can successfully implement the idea of the ‘Central Nervous System’, without spending many man-years of effort and still failing.</p><p><a href="https://github.com/specmesh/specmesh-build">Take a look at GitHub for more info</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bf057f4033a3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tools for Event-driven DevOps with Apache Kafka]]></title>
            <link>https://medium.com/@neil.avery_99/tools-for-event-driven-devops-with-apache-kafka-f4ca1721cc2a?source=rss-68d2da6ed2c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/f4ca1721cc2a</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[hackernoon]]></category>
            <category><![CDATA[event-driven-systems]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kafka-streams]]></category>
            <dc:creator><![CDATA[Neil Avery]]></dc:creator>
            <pubDate>Tue, 11 Feb 2020 11:35:53 GMT</pubDate>
            <atom:updated>2024-02-03T08:43:36.315Z</atom:updated>
            <content:encoded><![CDATA[<ul><li>Tools for Event-Driven DevOps with Apache Kafka</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5XbeBb4kjy44RakB4xHKkQ.jpeg" /><figcaption>“Empty highway road in through a colorful desert” by <a href="https://unsplash.com/@jplenio?utm_source=medium&amp;utm_medium=referral">Johannes Plenio</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>Update 2024 — Jan.</strong></p><p>Testing Kafka is hard — we have taken the well-used, and somewhat old kafka datagen and updating for SpecMesh testing, find updates here: <br><a href="https://github.com/specmesh/kafka-random-generator">https://github.com/specmesh/kafka-random-generator</a></p><p><strong>Update 2023.</strong></p><p>This post is now three years old. A lot has changed since then. I left Confluent, transitioned into freelancing, worked at a hedge fund, and have been in and out of more than 20 organizations that use Kafka, MSK, and CCloud. Regrettably, JulieOps, which had been gaining traction, is no longer in active development. Building on the experiences from large strategic engagements, platform build-outs, and other interesting projects, I, along with two others — Sion and Andy, started working on <a href="https://specmesh.io/">specmesh.io</a> (available on <a href="https://github.com/specmesh/specmesh-build">GitHub</a>). With complete bias, I believe it’s the right approach for adopting Apache Kafka and avoiding the common mistakes that make this industry somewhat challenging. Please take a look and feel free to reach out.</p><p><strong>Original Post</strong></p><p>DevOps is a long road; it is a journey. Are you Dev or Ops or both? This is the challenge I have struggled with as DevOps has become a thing. I’m part-way through writing a Kafka DevOps blog post for <a href="https://www.confluent.io/">Confluent</a> and this thought is always at the top of my mind. However, while I <a href="https://theagileadmin.com/what-is-devops/">grapple with it</a> — I figured I would provide a compiled list of Kafka DevOps tools we see in the wild.</p><p>An important point, much like the DevOps definition below, is that Kafka DevOps is not clearly defined; it is what you make of it. Each organization will have its own predispositions and DevOps culture. However, to properly embrace the event-driven architecture, there is a fundamental upheaval of existing concepts. Teams and functionality need to be structured differently, aligned to deliver business functionality, based upon dataflows and related bounded contexts. When one of the key benefits is a 24x7 run-time, and dynamic path execution (dataflow routing); a new feature is enabled using a feature flag; then we are winning. We can avoid the dreaded big bang release that fills you with fear of having to spend the rest of the weekend doing a rollback.</p><blockquote>What is DevOps Not? It’s Not NoOps. It’s Not (Just) Tools. It’s Not (Just) Culture. It’s Not (Just) Devs and Ops. It’s Not (Just) A Job Title. It’s Not Everything.<br><a href="https://theagileadmin.com/what-is-devops/">https://theagileadmin.com/what-is-devops/</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*G4I3z0XCW7mGC0C3S9Mdpg.png" /><figcaption>Some streaming applications can get complicated. Proper testing and ops strategies are essential</figcaption></figure><blockquote><a href="https://twitter.com/gAmUssA"><strong><em>Viktor Gamov</em></strong></a><strong><em> is also working on this — he just doesn’t know it yet ;)</em></strong></blockquote><blockquote><strong>Please leave your preferred tools and opinions in the comments section!</strong></blockquote><p><strong>Kafka Specific</strong></p><ul><li><a href="https://kafka.apache.org/11/documentation/streams/developer-guide/testing.html">Stream topology driver </a>(Java, Scala), Plus the <a href="https://github.com/jpzk/mockedstreams">MockedStreams</a> classes for the Processor API (streams only) (<a href="https://www.confluent.io/blog/stream-processing-part-2-testing-your-streaming-application/">Yeva provides a great introduction</a>)</li><li><a href="https://www.testcontainers.org/modules/kafka">TestContainers</a> (Java) for running brokers infra: ZooKeeper, Brokers and SchemaRegistry. Beware of the performance hit when spinning up containers</li><li><a href="https://github.com/FundingCircle/jackdaw/">Jackdaw</a> (Clojure) — <a href="https://www.confluent.io/blog/testing-event-driven-systems">from Funding Circle</a> — written by some very smart people. <a href="https://engineering.fundingcircle.com/blog/2017/11/26/testing-kafka-streams-topologies-with-kafka-interceptors/">A great article here</a></li><li><a href="https://github.com/lovoo/goka">GoKa</a>(Golang) Kafka Streams Impl</li><li><a href="https://github.com/nodefluent/kafka-streams">Nodefluent Kafka Streams</a> (Node) Kafka Streams</li><li><a href="https://github.com/authorjapps/zerocode/wiki/Kafka-Testing-Introduction">ZeroCode Kafka Testing</a> to produce and consume data (Java)</li><li><a href="https://github.com/RedPillAnalytics/gradle-confluent">Gradle Confluent plugin</a></li></ul><h3><strong>KSQL</strong></h3><ul><li><a href="https://docs.confluent.io/current/ksql/docs/developer-guide/api.html">KSQL-REST API</a></li><li><a href="https://gist.github.com/MichaelDrogalis/08463ed82a4a04015eef03cb483cad26">KSQL-CLI worked example</a> with std-in-out</li><li><a href="https://docs.confluent.io/current/ksql/docs/developer-guide/ksql-testing-tool.html">KSQL testing tool</a> Runs EmbeddedKafka with the topology test driver — fine for unit-tests but won’t support broader testing scenarios)</li></ul><h3>Infrastructure</h3><ul><li><a href="https://github.com/diegoicosta/kafka-confluent-examples/blob/master/kafka-streams/src/test/java/io/confluent/examples/streams/kafka/EmbeddedSingleNodeKafkaCluster.java">EmbeddedSingleNodeKafka</a> (Java or Scala unit test)</li><li>Debezium <a href="https://github.com/debezium/debezium/blob/master/debezium-core/src/test/java/io/debezium/kafka/KafkaCluster.java">Kafka Cluster utils</a> (Java or Scala)</li><li>Confluent docker-compose for broker-infra</li><li><a href="https://github.com/confluentinc/cp-ansible">Confluent Ansible</a></li><li><a href="https://www.confluent.io/confluent-operator/">Confluent Operator</a></li><li>CNCF <a href="https://strimzi.io/">Strimzi</a></li><li><a href="https://github.com/confluentinc/ducktape/wiki">DuckTape python-based, system test driver for Kafka</a></li><li><a href="https://github.com/confluentinc/muckrake">MuckRake — system tests on top of DuckTape</a></li><li><a href="https://github.com/Landoop/coyote">Connector testing using Coyote</a></li><li><a href="https://github.com/Landoop/kafka-testing">Kafka unit testing by Landoop</a></li><li><a href="https://medium.com/xebia-france/kafka-streams-a-road-to-autoscaling-via-kubernetes-417f2597439">Autoscaling steams</a> (might fit into later stages of a build pipeline)</li></ul><h3>Schema Registry integration</h3><ul><li>Schema Registry plugin for build tools</li><li>Confluent’s Maven Plugin <a href="https://docs.confluent.io/current/schema-registry/develop/maven-plugin.html">https://docs.confluent.io/current/schema-registry/develop/maven-plugin.html</a></li><li>SR Plugin for Gradle <a href="https://github.com/ImFlog/schema-registry-plugin">https://github.com/ImFlog/schema-registry-plugin</a></li></ul><h3>DataOps/generation</h3><ul><li><a href="https://github.com/edenhill/kafkacat">KafkaCat</a> generic non-JVM producer and consumer for Apache Kafka; think netcat for Kafka — by Magnus</li><li><a href="https://github.com/birdayz/kaf">Kaf</a> (like KafkaCat — Kafka CLI inspired by kubectl &amp; docker)</li><li><a href="https://www.confluent.io/product/ksql">KSQL</a> (Community license)</li><li><a href="https://lenses.io/">Lenses SQL</a></li><li><a href="https://github.com/Interana/eventsim">EventSim</a> for data generation of simulated data</li><li><a href="https://docs.confluent.io/current/ksql/docs/tutorials/generate-custom-test-data.html">Kafka Avro datagen</a> simulation tool</li></ul><h3><strong>General</strong></h3><ul><li>Unit testing shell scripts with <a href="https://github.com/kward/shunit2/">shunit2</a>.sh: <a href="https://www.leadingagile.com/2018/10/unit-testing-shell-scriptspart-two/">https://www.leadingagile.com/2018/10/unit-testing-shell-scriptspart-two/</a> <a href="https://www.leadingagile.com/2018/10/unit-testing-shell-scriptspart-one/">(part 1 is interesting)</a></li><li><a href="https://github.com/kward/shunit2/">Shunit2</a> with Maven : <a href="https://leelevett.wordpress.com/2015/07/21/bash-script-project-with-tests-and-maven/">https://leelevett.wordpress.com/2015/07/21/bash-script-project-with-tests-and-maven/</a></li><li>Shunit2 with TravisCI : <a href="https://www.sysorchestra.com/travisshunit2/">https://www.sysorchestra.com/travisshunit2/</a></li><li>Sample github project that uses TravisCI to test a .sh script : <a href="https://github.com/martinseener/example-shunit2-travisci">https://github.com/martinseener/example-shunit2-travisci</a> AND <a href="https://github.com/soulseekah/test-shunit2-travis">https://github.com/soulseekah/test-shunit2-travis</a></li><li>TravisCI support for Docker : <a href="https://blog.travis-ci.com/2015-08-19-using-docker-on-travis-ci/">https://blog.travis-ci.com/2015-08-19-using-docker-on-travis-ci/</a></li></ul><h3>What’s missing</h3><p>Quite a lot of stuff is missing, but that’s because most environments will be geared towards a particular deployment and automation runtime.</p><ul><li>Data tooling — topic provisioning, topic pre-population (from source or flat file), topic-assertion-validation</li><li>Infrastructure automation tools (Ansible, Helm, Helm charts, Operators)</li><li>Schema/quota/governance automation via APIs and assertions</li><li>Security/ACL automation (Very important)</li></ul><h3>Resources</h3><ul><li><a href="https://dzone.com/articles/a-quick-and-practical-example-of-kafka-testing">DZone: A quick and practical example of Kafka testing.</a></li><li><a href="https://medium.com/bakdata/fluent-kafka-streams-tests-e641785171ec">Fluent Kafka Streams</a></li><li><a href="https://www.confluent.io/blog/deploying-kafka-streams-and-ksql-with-gradle-part-3-ksql-user-defined-functions-and-kafka-streams">Deploying Kafka Streams with KSQL and Gradle — UDFs and more</a></li></ul><p><strong>Want to learn more?</strong></p><ul><li><a href="https://medium.com/@neil.avery_68603/journey-to-event-driven-part-1-why-event-first-thinking-changes-everything-c98b98e248d5">Journey to Event-Driven — Part 1: Why Event-First Thinking Changes Everything</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f4ca1721cc2a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Quarkus, TestContainers, DynamoDB and CircleCI]]></title>
            <link>https://medium.com/@neil.avery_99/using-quarkus-testcontainers-dynamodb-and-circleci-5c41e8702ad0?source=rss-68d2da6ed2c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/5c41e8702ad0</guid>
            <category><![CDATA[circleci]]></category>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Neil Avery]]></dc:creator>
            <pubDate>Fri, 07 Feb 2020 12:36:07 GMT</pubDate>
            <atom:updated>2020-02-07T12:36:07.391Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*lsP46_Tc2GhxvugpASRXJQ.png" /><figcaption>Quarkus for goodness — meta-style java development with GraalVM</figcaption></figure><p>As part of the <a href="https://github.com/liquidlabsio/precognito">Precognito</a> github project (<em>details soon to be set forth — yes it has some machine learning</em>), I’ve been setting up the github project to use CircleCI. One of the goals of the project is to adopt best of breed technology and build a completely serverless log analysis platform that runs in your VPC — in the free tier ;) It is using Quarkus (because my twitter friends cannot stop talking about it), CircleCI and serverless cloud technologies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/310/1*ViiMb70mR3sCIVNrBL0KWw.png" /></figure><p>I hit a few barriers along the way, some of which are my own lack of understanding or misconceptions about how to get the ‘new’ stuff working. I’ve tried to capture each item below.</p><p><strong>TestContainers and DynamoDB local for testing</strong></p><p>If you haven’t looked at or learned about TestContainers then you are missing out ;). It is definitely something to have in your arsenal. While fixtured APIs are ideal (and fast) there will come a time where you need Dockerized services to integration test against. I say integration test, because using Docker in testing is slow, but kind of fully featured. Below is a list of issues I hit.</p><ol><li>TestContainers uses docker commands and requires ‘authorized configuration’ otherwise you get a nasty error like: <em>‘docker-credential-desktop’ not found</em>. This is because <strong>.docker/config.json</strong> is telling it to ‘auth’ itself. Remove the line <em>“credsStore” : “desktop”</em> — but be aware of why the line exists and if you should indeed remove it!</li><li>TestContainers provides a <a href="https://www.testcontainers.org/modules/databases/dynalite/">dynalite</a> localised dynamodb instance (a dynamodb alternative). However, it is simply a wrapper over the TestContainers GenericContainer. Quarkus advocate the semi-experimental <strong>dynamodb2</strong> <a href="https://github.com/awsdocs/aws-doc-sdk-examples/tree/master/javav2/example_code/dynamodb/src/main/java/com/example/dynamodb">client driver</a>. The dynalite tescontainer provides a client builder for the old version, so instead adopted my own — see <a href="https://github.com/liquidlabsio/logscape-ng/blob/master/uploader/src/test/java/com/logscapeng/uploader/aws/LocalDynamoDbContainer.java">here</a>. I could then switch between different docker images to see which one suited my needs. I ended up using the aws/dynamo-local it seemed to work better with the new v2 client drivers.</li><li>To speed up testing you don’t want to start/stop Docker all the time. Using a Junit5/Juniper @BeforeAll helps, however, the standard approach is to declare required resources as static variables. Due to Quarkus CDI injection, it was somehow classloading the unit-test static fields (wha??). This made non-related unit tests spin up <a href="https://www.testcontainers.org/">testcontainers</a> — I could see it in the log output — and was like WTF? To work around this problem, rather than following what TestContainers advocate (see <a href="https://www.testcontainers.org/test_framework_integration/junit_5/">here</a>) I simply create it in the @BeforeAll static function — as shown <a href="https://github.com/liquidlabsio/logscape-ng/blob/master/uploader/src/test/java/com/logscapeng/uploader/DynamoIndexingTest.java#L32">here</a>.</li></ol><p><strong>Test Containers and CircleCI</strong></p><p>Upon checking-in I soon realized that TestContainers would not work with the default CircleCI dockerized <a href="https://circleci.com/docs/2.0/executor-types/">executor.</a> This would mean running docker within docker == FAIL! I had to switch to the ‘Machine’ executor. This means updating <strong>.circleci/config</strong> to not only specify the execution type, but also the maven build steps. The config file is <a href="https://github.com/liquidlabsio/logscape-ng/blob/master/.circleci/config.yml">here</a>. I will eventually get time to setup and build the GraalVM native image. I followed these docs on building <a href="https://quarkus.io/blog/quarkus-app-circleci-build/">Quarkus on CircleCI</a>.</p><p><strong>Quarkus and application profiles</strong></p><p>Application profiles are a really nice feature of Quarkus. It means I can run unit or integration tests in the IDE, commandline or CI build and have a specific behavior — I might want to use fixtures for super fast local builds, and docker images for CI integration builds. By default, Quarkus supports <em>test</em>, <em>dev</em> and <em>production</em> profiles. The config values are held in an <a href="https://github.com/liquidlabsio/logscape-ng/blob/master/uploader/src/test/resources/application.properties">application.properties</a> file. The test/resources files only need to contain test values, however, the shipped <a href="https://github.com/liquidlabsio/logscape-ng/blob/master/uploader/src/main/resources/application.properties">properties</a> file will specify other profiles. I use this profiling to control when fixtures are loaded, to this I set up a <a href="https://github.com/liquidlabsio/logscape-ng/blob/master/uploader/src/main/resources/META-INF/services/org.eclipse.microprofile.config.spi.Converter">microprofile convertor config</a> to allow the various classes to be specified so the CDI injection loads the right thing! The only gotcha here was that I needed my application runtime to sometimes know when profile is being run. Im still not 100% certain that its the right approach. You will see here that the <a href="https://github.com/liquidlabsio/logscape-ng/blob/master/uploader/src/main/java/com/logscapeng/uploader/aws/AwsStorageConverter.java">AWSStorageConvertor</a> can return a fixture when in test mode.</p><p>There is a fairly basic web-interface so feel free to grab it download. It is possible to run it locally with ‘maven quarkus:dev’ — in which case it uses fixtured cloud services!</p><p>All the best!</p><p>Twitter @avery_neil</p><p><a href="http://www.liquidlabs.com/">www.liquidlabs.com</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5c41e8702ad0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Journey to Event Driven — Part 1: Why Event-First Thinking Changes Everything]]></title>
            <link>https://medium.com/@neil.avery_99/journey-to-event-driven-part-1-why-event-first-thinking-changes-everything-c98b98e248d5?source=rss-68d2da6ed2c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/c98b98e248d5</guid>
            <category><![CDATA[enterprise-technology]]></category>
            <category><![CDATA[event-driven-systems]]></category>
            <category><![CDATA[event-driven-architecture]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[startup]]></category>
            <dc:creator><![CDATA[Neil Avery]]></dc:creator>
            <pubDate>Fri, 07 Feb 2020 10:33:54 GMT</pubDate>
            <atom:updated>2020-02-12T11:37:58.888Z</atom:updated>
            <content:encoded><![CDATA[<h3>Journey to Event-Driven — Part 1: Why Event-First Thinking Changes Everything | Apache Kafka</h3><p><em>Note, this is an updated version since it was originally published on the Confluent blog (</em><a href="https://www.confluent.io/blog/journey-to-event-driven-part-1-why-event-first-thinking-changes-everything/"><em>link</em></a><em>)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NAZSv7xu1ihEDrachGpnfg.jpeg" /><figcaption>Image by <a href="https://pixabay.com/users/JanBaby-3005373/?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=1556177">Jan Alexander</a> from <a href="https://pixabay.com/?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=1556177">Pixabay</a></figcaption></figure><p>The world is changing. New problems need to be solved. Companies now run global businesses that span the globe and hop between clouds, breaking down silos to create seamless applications that synergize the organization. This continuous state of change means that legacy architectures are insufficient or unsuitable to meet the needs of the modern organization. Applications must be able to run 24×7 with 5–9s (uptime of 99.999%), as well as be super-elastic, global and cloud-native.</p><p>Traditional architectures simply cannot meet the challenges of real-time and extreme scale. They have no mechanical sympathy and simply won’t work. It’s often said and well-proven throughout the NoSQL era that systems are designed to suit particular types of workloads and use cases. A time-series database isn’t very good at handling relational use cases or modeling, and likewise, a document database is great at providing structure, but not so good at analytics against those documents.</p><p>It sounds like a stretch of the imagination that any architecture could solve such extreme challenges. But fundamentally speaking, today, we are addressing these new, rising needs through microservices, IoT, event hubs, cloud, machine learning and more.</p><p>At some point, it becomes obvious that we need to go back to basics, back to first principles of system design, and start again. The common element of all these new world problems is that they revolve around the notion of events. These events drive actions and reactions, and transform between different streams, splitting, merging and evolving like the pathways of your brain.</p><p>To understand the importance of being event-driven, we’ll examine why events have become so pivotal in our thinking today. We will then evaluate the qualities and how events have become a first-class concern for the modern organization, as awareness of events underpins event-first thinking and design. I’ll then extrapolate design concepts against different perspectives of event patterns, such as the event command and event first, and why the event-first approach fulfills foundational requirements when choosing an architecture that supports evolution and elasticity.</p><h3>Overview</h3><ul><li><a href="https://www.confluent.io/blog/journey-to-event-driven-part-1-why-event-first-thinking-changes-everything/#why-events-matter">An opinionated history of “events” — Why do they matter?</a></li><li><a href="https://www.confluent.io/blog/journey-to-event-driven-part-1-why-event-first-thinking-changes-everything/#event-adoption-journey">Adoption journey of the “event”</a></li><li><a href="https://www.confluent.io/blog/journey-to-event-driven-part-1-why-event-first-thinking-changes-everything/#event-driven-architecture">Considerations of the event-driven architecture</a></li><li><a href="https://www.confluent.io/blog/journey-to-event-driven-part-1-why-event-first-thinking-changes-everything/#event-first-thinking">Transitioning to event-first thinking</a></li><li><a href="https://www.confluent.io/blog/journey-to-event-driven-part-1-why-event-first-thinking-changes-everything/#event-driven-design">Event-first versus event-command patterns for event-driven design</a></li><li><a href="https://www.confluent.io/blog/journey-to-event-driven-part-1-why-event-first-thinking-changes-everything/#event-command-pattern">Event-command pattern</a></li><li><a href="https://www.confluent.io/blog/journey-to-event-driven-part-1-why-event-first-thinking-changes-everything/#event-first-cost">The cost of the event-first approach</a></li><li><a href="https://www.confluent.io/blog/journey-to-event-driven-part-1-why-event-first-thinking-changes-everything/#event-first-benefits">Benefits of the event-first approach</a></li></ul><h3>An opinionated history of “events” — Why do they matter?</h3><p>It’s taken the last 25 years to get to the point where we consider the event as a core principle in system design. Throughout that period, much of the world was stuck thinking that remote object calling was the right way of doing things. REST and web services started by exposing the API as a concern, but that was done by a synchronous protocol.</p><p>In 2002, I built a commercial messaging system that adopted principles from the <a href="http://www.sosp.org/2001/papers/welsh.pdf">SEDA: An Architecture for Well-Conditioned, Scalable Internet Services paper</a> — the idea being that we move everything to become asynchronous; local and remote events are then used to drive the intelligence and adaptation of an application. It really shaped the way I understood events to be a core design principle as well as how to think about system design and data in general.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*RK0iY9Aphc9YwhgZ.png" /></figure><blockquote>There has been a revolution; organizations must become real time; to become real time, they must become event driven</blockquote><p>It took the era of big data for us to realize that while having lots of data is useful, things get slower as you have more of it. And yet, we collect more. Over the last several years, there has been a revolution; organizations must become real-time; to become real-time, they must be event-driven. This need is fueled by the increasingly digital world that we live in where everything is connected. Events are a fundamental aspect that drives this ecosystem; an organization is but a cog that when viewed at any level needs to act and react to internal and external events.</p><p>The value of events is that a sequence of related events represent behavior (e.g. an item was added and then removed from a shopping cart, an error recurs every 24 hours or users always clicks through a site in a particular order). A sequence of related events is commonly called a <em>stream</em>. Streams can come from an IoT device, a user visiting a website, changes to a database or many other sources. When thinking of APIs, we start with the <em>event</em> rather than the highly coupled concept of a ‘command’, e.g., an <em>event-X</em> has <strong>occurred</strong>, rather than <em>command-Y</em> should be <strong>executed</strong>. This thinking underpins event-driven streaming systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/0*y0At6G5ga4lkU0Fo.png" /><figcaption>A sequence of events captures a story. Replaying the events allows time-travel, to see that behavior again.</figcaption></figure><h3>Adoption journey of the “event”</h3><p>There are many paths to the inception of an event-driven architecture within an organization. For some, it is a natural pattern evolving from <a href="https://en.wikipedia.org/wiki/Domain-driven_design">domain-driven design (DDD)</a> principles, the related and powerful event-storming practice, decoupled systems design or just event-driven microservices. The point is they can come from any level, whether business or technical, but it will be a conscious adoption that doesn’t just happen by accident.</p><h3>Considerations of the event-driven architecture</h3><p>There are many considerations when evaluating the event-driven architecture. Events initially start out to as atomic and drive reactionary callbacks (functions). As a use case evolves, workflows develop and the criticality of the application generally increases along with the nature in which an event is considered. This raises numerous questions. Is ordering important? Do I want to ensure transactionality? How do I trust the execution? Security? Lineage? Where did the event come from?</p><blockquote>The key realization for the adoption of event-first thinking is that an event represents a fact, something happened; it is immutable and therefore changes how we think about our domain model</blockquote><p>The basic attributes of an event include time, source, key, header, metadata, and payload. One problem with events is that they are everywhere, and the extreme lack of consistency makes them very difficult to work with at scale. In an effort to solve this, <a href="https://cloudevents.io/">CloudEvents 1.0</a> was released by the Cloud Native Computing Foundation (CNCF) in October 2019.</p><p>A CloudEvent is a standard event envelope containing sufficient information for any consumer to use correctly (schema, source, etc). The most interesting note is that the <a href="https://github.com/cncf/wg-serverless">CNCF Serverless Working Group</a> is standardizing the serverless model to encompass how CloudEvents work with FaaS functions, function signatures, workflow, extensions, transport bindings, SDKs and more. The future of serverless will be one where FaaS functions can work together across cloud boundaries, the enterprise and all corners of the globe. It is pretty exciting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/0*CKj3niP4rISdTz9S.png" /></figure><p>The other consideration is that events do not exist in isolation. Due to their very nature, an event tends to be part of a flow of information, a stream. We define a stream as an unbounded sequence of related events that are associated through the use of an “eventKey,” a key that ties multiple events together.</p><p>In the real world, the nature of the “event” can be described as:</p><ul><li>Atomic: something happened (bid on an item, send an email, device temperature)</li><li>Related: a stream or sequence of events (tracking a pricing change, device metrics changes over time)</li><li>Behavioral: the accumulation of facts captures behavior</li></ul><p>Behavioral is the least obvious but most important aspect of event-first thinking. For example, in a shopping cart, if an item was added and removed — and this has occurred among many users — then we have captured a behavior. By analyzing the sequence across many users, it is possible to discern whether they changed their mind due to targeted advertising, finding a cheaper price elsewhere or perhaps because it is close to an annual event like Christmas, a birthday, etc. Ideally, the system should detect the behavior and react to it, as capturing events has a great deal of implicit value.</p><h3>Transitioning to event-first thinking</h3><blockquote>Event-first thinking changes how you think about what you are building</blockquote><p>The adoption of event-driven architectures tends to result from an “aha” moment, which involves moving from simple, use cases to a deeper understanding of processes, operations and how to reason about the architecture.</p><p>You may start off creating the event streams required to build an application to fulfill your original design goals, but quickly find that the event streams themselves enable you to build much richer, totally unexpected analysis and functionality.</p><p>How is the event treated within a system? All of the following questions get asked as the event-driven architecture is developed:</p><ul><li>Is it observable, and are the flows of streams behaving as expected?</li><li>Is it trusted, meaning transactional, exactly one or at least once? Will it scale?</li><li>Is stateless processing, such as filtering, projection, cleaning or enrichment required? Is stateful processing, such as aggregations or stateful sequence processing required?</li><li>Is a materialized view against the stream required? How many transactions per second are required for a windowed view?</li><li>Do we want to scale/fan/map out (parallelize), fan-in/collect and, build materialized views?</li><li>Does it support error handling, such as error flows and dead letter queue?</li><li>Does it send and transform events from one stream into others (stream processing)?</li><li>Does it react and drive intelligence from the state collected from a stream (stream processing)?</li></ul><p>In terms of increased adoption, we generally see these patterns as characteristic of an event-driven organization:</p><ul><li>Data pipelines (ETL, integration)</li><li>Monitoring and alerting, including log collection and aggregation</li><li>Event-driven microservices</li><li>Enterprise wide event-driven microservices</li><li>IoT</li><li>Customer 360</li><li>Digital transformation</li><li>Real-time enterprise</li><li>Central nervous system/digital nervous system (using CloudEvents)</li></ul><p>By embracing event-first thinking, we naturally inherit the foundations of event streaming platforms like event sourcing, replayability, stream processing, and dataflow design, amongst others. We should also incorporate event-first thinking into an architecture that leverages a serverless stack to be event-driven, multi-cloud and elastic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/0*89mZ_wPfncjxL8tI.png" /></figure><p>There are several ways to be <em>event-driven</em>. Without becoming tangled up in the marketing noise, we can boil it down to a couple of approaches. The first uses messaging principles and sends pure events like item-purchased while the second reuses the conventional, more familiar approach of sending commands as events, like user.purchaseItem(item). Both can be event-driven, but the design principles are fundamentally different. Let’s expand on this thinking to understand the impact of both approaches.</p><p>The importance of events and event first thinking:</p><ul><li>Capture facts</li><li>Capture behavior</li><li>Provide a representation of the real world</li><li>Model use cases of how we think</li><li>Supports repeated evaluation and processing (a time machine)</li><li>Provide horizontal scaling</li><li>Speak the same language as the business (ubiquitous language)</li></ul><h3>Event-first versus event-command patterns for event-driven design</h3><p>You can build the same application using either the event-first or event-command approach, but the event-first approach is generally recommended. If we rewind onto familiar ground for an example, we see that traditional REST applications are an event → command pattern. They are coupled in multiple ways: Firstly, the endpoint is known (i.e., the service address). Secondly, the method being called is also known (i.e., an API call to doStuff), and lastly, the calls tended to return a value; they are synchronous. Newer frameworks will allow this call to be handled synchronously or asynchronously — it is called the event-command pattern.</p><p>Alternatively, with the event-first pattern, the paradigm shift is to discard all of those considerations and just send an event as we would in the traditional messaging sense; don’t do anything else; have no API coupling to a remote service. The approach is unique in that it processes the event as a reaction; the emitter doesn’t call on a specific function; the API has been removed and it instead just sends an event. The emitter of the event doesn’t know which processors (or functions) are going to consume it, and the event becomes the API. This decoupling allows the set of consuming apps to change over time without any upstream changes required in the emitter.</p><p><strong>Event-first analog:</strong> I walk into a room, generate an “entered room” event and the light turns on. This is a reaction to an event.</p><p><strong>Event-command analog:</strong> I walk into a room, <em>flip the light switch</em> and the light turns on. This is a command.</p><p>In the event-first analog, I have no knowledge and don’t ask for the lights to be turned on; instead, a sensor detects my presence; it has the responsibility. In the event-command analog, I have the responsibility of knowing how to turn on the light and also making it happen.</p><blockquote>The event-first approach forces an inversion of responsibility; it is a fundamental paradigm shift of how applications are developed</blockquote><p>Let’s apply it to a business function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/0*D8_Wx2M89BLBkAvs.png" /><figcaption>The Purchase processor knows what to call and in which sequence; high coupling makes architectural evolution difficult</figcaption></figure><h3>Event-first pattern</h3><p>With the event-first approach, each service listens for inbound events and outputs zero or more events after performing some (re)action. For example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/0*r06ixFz_sx0mJvDk.png" /><figcaption>Everything emits its event; easy to replace with new processors and support change</figcaption></figure><p>At this point, the two approaches don’t look that different. If anything, the event-first approach looks more complex, as it has to more moving parts, e.g., publishing/consuming events via streams. The benefit is that these streams provide decoupling.</p><p>The power of the event-first approach becomes clear when we try to enhance the system. Let’s say we want to add the ability to check inventory levels and reorder stock when levels are low, to perform analysis on the cause of failed purchases and help improve conversion rates.</p><p>For the event-command-based system, we need to change the main purchasing application to call these new systems. But for the pure event-first solution, we can simply add new processors of the existing events, such as why(item) and checkInventory().</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/0*S089WXkE1Sh-4OoT.png" /><figcaption>Events provide low-coupling; adding new functionality is done without shutting the system down. Functionality can be enabled using feature flags</figcaption></figure><p>Notice how the event-driven approach allows many processors to react to the same “purchased” event (published via the /user-purchases topic)? This flexibility means functionality can run in parallel and be added, extended, upgraded or replaced without taking the system down. Decoupling is the key. The “event” also affords the opportunity to propagate versioning, security tokens, correlation IDs and other useful information.</p><h3>The cost of the event-first approach</h3><p>The tradeoff of event-first/event-driven architecture is the amount of scaffolding required to inspire confidence. In the old days when <a href="https://en.wikipedia.org/wiki/Consistency_(database_systems)">transactional consistency</a> shaped our expectations, and NoSQL pushed us to eventual consistency with <a href="https://en.wikipedia.org/wiki/CAP_theorem">CAP theorem</a> tradeoffs, it was known that data wouldn’t (and shouldn’t) get lost. Event orientation requires not only different thinking in replacing the command model but also the ability to support traceability, failure paths, scaling and explanation about why things have gone wrong. These challenges are easily solvable and are also quite fun; I will revisit this topic later on.</p><p>The other cost is ‘scale’ and dealing with complexity: you need to leverage sensible mechanisms to avoid the dreaded — pin-ball architecture (new term). In short, this is done by using hierarchical topic naming, a governance model, and ensuring that events-as-an-API is embraced. Event streams within a bounded context should be hidden. Only those at the entry/exit points should be available.</p><h3>Benefits of the event-first approach</h3><p>The pure event-first approach described above demonstrates:</p><ul><li><strong>Decoupling:</strong> Processors don’t know anything about upstream or downstream processors</li><li><strong>Encapsulation:</strong> There are clean boundaries between processors.</li><li><strong>Inverted knowledge:</strong> Knowledge and responsibility are reversed.</li><li><strong>Evolutionary change:</strong> The system and events can change over time.</li><li><strong>Event sourcing:</strong> When using a log and log-aware stream processors, we gain the ability to potentially rebuild and restore application state. It’s not a free lunch; the application state needs to be captured and managed. We don’t want to have to re-build event derived state by replaying events from the beginning of time.</li></ul><blockquote>Evolutionary architecture is a natural benefit from event-first thinking and event-driven architectures</blockquote><p>When you use the pure, event-driven approach, the architecture can change over time as different processors may react to events, which can be reprocessed while the data model evolves simultaneously. Perhaps the domain model has changed, or functionality is being extended. None of the originating events are lost, and the behavior is captured in the stream. With these foundations, you then have a system that supports architectural evolution as well as data and domain evolution.</p><p>Rebecca Parson, Neal Ford and Patrick Kua also wrote a great book on these principles called <a href="https://www.thoughtworks.com/books/building-evolutionary-architectures">Building Evolutionary Architectures</a>. If you’d like more background reading, Martin Fowler’s blog post <a href="https://martinfowler.com/eaaDev/EventNarrative.html">Focusing on Events</a> paints a nice story too (albeit it was written in 2006).</p><p>To support the event-first approach you need a suitable platform. Evaluating the pros and cons, knowing where to start and understanding the ecosystem and everything else that goes with that decision is in and it of itself a complete mission. There are a plethora of blogs and vendors creating noise that crowds and confuses the market.</p><p>I personally deplore anything that resembles an opinioned framework; they are the pinnacle of IT gone wrong (too much magic). I’ve spent too many years upgrading from one framework to another only to be stung along the migration path.</p><p>The ideal solution is one where the stack is simple, the surface area is small and ‘I’ the developer retain control. This normally dictates some form of the programming model, and as distributed computing moves to its next-generation, we also need to choose something that is forward-looking. It must fit with the types of distributed applications that we need to build for tomorrow and not constitute a retrofit of old-school RPC.</p><h3>Orchestration versus Choreography</h3><p>I will touch on this topic briefly, but it is due much more space than I have allowed. The event-first architecture is described as Choreography. Events don&#39;t know who will consume them, they just exist; there is no master controller. This principle instills evolution, supports feature flags at runtime where new functionality is enabled without taking the application down (flags are used to determine where events are consumed). Orchestration is a workflow, the relationship between events is owned by a third party. The third-party is responsible for routing, filtering, but also failure and recovery scenarios. If a processor fails to handle an event, it knows to undo-previous states on upstream processors. With Choreography, you need to use a sensible approach to recovery (<a href="https://www.goodreads.com/en/book/show/23463279-designing-data-intensive-applications">Martin Kleppman explains it nicely in his excellent book</a>). Using the <a href="https://microservices.io/patterns/data/saga.html">Saga pattern</a> to handle failures offers an understood approach. There will be situations that Orchestration is preferred over Choreography and vice-versa; they are not mutually exclusive, yet instead, will co-exist in the large event-driven organization. Choreography is ideal for connecting the organization together at scale, <a href="https://riccomini.name/how-paint-bike-shed-kafka-topic-naming-conventions">hierarchical topic naming</a> keeps topic organized and allows governance models to be put in place.</p><h3>Finally…</h3><p>I have reasoned about how event-first thinking represents a new opportunity, a new frontier for technology and businesses to go back to first principles of system architecture and design. Changing the underlying foundations of how we conceive technology naturally shapes how we approach business problems, and therefore how we develop technology.</p><p>This revaluation on technical design pushes us towards a pure eventing model where we throw out the event-command pattern and simply emit events. The benefit of this approach meets many modern requirements where systems are elastic, easily composable, expected to run 24×7 and, most importantly, are evolvable. The ability to progress domain models that are supported by events and remodel event streams is an essential requirement for the modern business.</p><p>It’s not a free lunch, however. With the event-first model comes the need to build trust and confidence that the system not only scales but also functions as expected, which is kind of important, right?</p><p>I hope you join me in <a href="https://www.confluent.io/blog/journey-to-event-driven-part-2-programming-models-event-driven-architecture">part 2</a> of this series, where I’ll walk through different styles of event-driven architectures. Choosing the right style of architecture is as important as the realization of becoming event-driven, with each approach having various pros and cons. The conclusion isn’t surprising, but the world is changing, and we need to understand technology through a new lens. Just like learning a new programming language, we need to try and push our predispositions aside, and embrace the anarchy of thinking and rationale!</p><h3>Interested in more?</h3><p>If you are based in London and want to meet for a coffee or deep-dive into your solution architecture — give me a shout at my new home: <a href="https://liquidlabs.com/">liquidlabs.com</a></p><p>In episode 18 of the Streaming Audio podcast, I also discuss <a href="https://www.buzzsprout.com/186154/954362-the-future-of-serverless-and-streaming-with-neil-avery">the future of serverless and streaming</a> with Tim Berglund.</p><p>If you’d like to know more, you can <a href="https://www.confluent.io/download/">download the Confluent Platform</a>, the leading distribution of Apache Kafka® and run through the <a href="https://docs.confluent.io/current/quickstart/index.html">quick start</a>.</p><p>Other articles in this series:</p><ul><li><a href="https://medium.com/@neil.avery_68603/journey-to-event-driven-part-1-why-event-first-thinking-changes-everything-c98b98e248d5">Journey to Event-Driven — Part 1: Why Event-First Thinking Changes Everything</a></li><li><a href="https://medium.com/@neil.avery_68603/journey-to-event-driven-part-2-programming-models-for-the-event-driven-architecture-eded219630a2">Journey to Event-Driven — Part 2: Programming Models for the Event-Driven Architecture</a></li><li><a href="https://medium.com/@neil.avery_68603/journey-to-event-driven-part-3-the-affinity-between-events-streams-and-serverless-apache-84bd37d0f9b9">Journey to Event-Driven — Part 3: The Affinity Between Events, Streams, and Serverless</a></li><li><a href="https://medium.com/@neil.avery_68603/journey-to-event-driven-part-4-four-pillars-of-event-streaming-microservices-apache-kafka-9e88fd520a07">Journey to Event-Driven — Part 4: Four Pillars of Event Streaming Microservices</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c98b98e248d5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Journey to Event-Driven — Part 4: Four Pillars of Event Streaming Microservices | Apache Kafka]]></title>
            <link>https://medium.com/@neil.avery_99/journey-to-event-driven-part-4-four-pillars-of-event-streaming-microservices-apache-kafka-9e88fd520a07?source=rss-68d2da6ed2c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e88fd520a07</guid>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[event-driven-systems]]></category>
            <category><![CDATA[event-driven-architecture]]></category>
            <category><![CDATA[kafka-streams]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Neil Avery]]></dc:creator>
            <pubDate>Thu, 09 May 2019 17:00:20 GMT</pubDate>
            <atom:updated>2020-02-11T13:05:23.230Z</atom:updated>
            <content:encoded><![CDATA[<h3>Journey to Event-Driven — Part 4: Four Pillars of Event Streaming Microservices | Apache Kafka</h3><p><em>Please note, this is an updated version since it was originally published on the Confluent blog; many things have changed! (</em><a href="https://www.confluent.io/blog/journey-to-event-driven-part-4-four-pillars-of-event-streaming-microservices"><em>link</em></a><em>)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v1YusRswRcxKCXg46cjLuw.jpeg" /><figcaption>Image by <a href="https://pixabay.com/users/LoggaWiggler-15/?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=2973">LoggaWiggler</a> from <a href="https://pixabay.com/?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=2973">Pixabay</a></figcaption></figure><p>So far in this series, we have recognized that by going back to first principles, we have a new foundation to work with. Event-first thinking enables us to build a new atomic unit: the event. Storing events in a stream and connecting streams via stream processors provide a generic, data-centric, distributed application runtime that you can use to build ETL, event streaming applications, applications for recording metrics and anything else that has a real-time data requirement.</p><p>This model is completely free form, we can build anything provided that we apply <a href="https://groups.google.com/forum/#!forum/mechanical-sympathy">mechanical sympathy</a> with the underlying system behavior. It offers time travel, a source of truth, scaling, consistency and many elements that we can rely on. All of these, and more, lead to design patterns that can be used and reused.</p><blockquote>Everything is asynchronous; everything is stored; everything is partitioned; everything scales</blockquote><h3>Overview</h3><ul><li>Four pillars of event streaming</li><li>Pillar 1 — Business function: Payment processing pipeline</li><li>Pillar 2 — Instrumentation plane: Business metrics</li><li>Pillar 3 — Control plane: Flow control, start, pause, bootstrap, scale and rate limit</li><li>Pillar 4 — Operational plane: Event logging, DLQs and automation</li><li>Deployment model</li><li>Parting thoughts and preparing for the future</li></ul><p>To read the other articles in this series, see:</p><ul><li><a href="https://medium.com/@neil.avery_68603/journey-to-event-driven-part-1-why-event-first-thinking-changes-everything-c98b98e248d5">Journey to Event Driven — Part 1: Why Event-First Thinking Changes Everything</a></li><li><a href="https://medium.com/@neil.avery_68603/journey-to-event-driven-part-2-programming-models-for-the-event-driven-architecture-eded219630a2">Journey to Event Driven — Part 2: Programming Models for the Event-Driven Architecture</a></li><li><a href="https://medium.com/@neil.avery_68603/journey-to-event-driven-part-3-the-affinity-between-events-streams-and-serverless-apache-84bd37d0f9b9">Journey to Event Driven — Part 3: The Affinity Between Events, Streams and Serverless</a></li></ul><p>It is not uncommon for those embarking on the journey to event-driven, asynchronous architecture to ask:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*25MdpnmstlnYftOc.png" /></figure><p><em>“Hey, so I’m writing these events, how do I know what’s happening and whether they’re working as intended? I don’t trust my application when it is asynchronous-I really want transactions.”</em></p><p>Let’s explore what this really means. We need to build something real, such as a payment system. <em>Why a payment system, you ask?</em> Well, pretty much every business out there needs to get paid, so it seems like a very common use case. It is easy to understand but also scary for developers who must trust an action that moves money from one account to another only using a transaction.</p><p>Transactions give you all or nothing (atomic) behavior. They provide a guaranteed level of trust; they are simple. When we do replace transactions with events, it can seem complicated. Yes, we can <a href="https://www.confluent.io/blog/transactions-apache-kafka/">transact across partitions</a> in Apache Kafka ®, and with traditional database transactions you get many “trust” semantics for free; however, they are pushed down into the database runtime. The problem is that many traditional OLTP databases are also relatively slow when they need to scale. 😉</p><p>Not only does the event streaming model expose the developer to building transactional semantics, scaling and error handling, etc., but it also means that they have more control, a richer ability to extend and enrich the system. Although this sounds like it adds complexity, the opposite is true. Following the introduced patterns and working within <a href="https://martinfowler.com/bliki/BoundedContext.html">bounded contexts</a> it actually contains the complexity within each context.</p><p>We also have the opportunity to add layers of functionality to the payment system as it evolves. Some banks run up to 30 processing stages, which stretches well beyond the realms of a single, atomic transaction, and pushes the complexity of <a href="https://en.wikipedia.org/wiki/Distributed_transaction">XA transactions</a> -which don’t exist in the event-driven world.</p><blockquote>“We believe that the major contributor to this <strong>complexity</strong> in many systems is the <strong>handling of state</strong> and the burden that this adds when trying to analyse and reason about the system” [emphasis added]. <br><a href="https://github.com/papers-we-love/papers-we-love/blob/master/design/out-of-the-tar-pit.pdf">Out of the Tar Pit, 2006</a></blockquote><p>The most challenging goal of any application architecture is <strong>simplicity</strong>; but it is possible to achieve. I’m going to explore four pillars for enabling scalable development that works across the event-driven enterprise. These pillars minimize complexity and provide foundational rules for building systems using composition. To wrap it up, I’ll touch on the natural progression of the payment system, including enterprise adoption, events as a backbone and DevOps.</p><h3>Four pillars of event streaming</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*zzquzEActcOlWUNY.png" /><figcaption><em>Recognizing architectural pillars and patterns allows us to move away from complexity.</em></figcaption></figure><p>To adopt event-first thinking, we need to stop, take a breath and deeply understand the world of transaction processing, which we often take for granted, and then map its mechanics to the event-driven streaming world. (Note: We view event streaming as the next generation of event-driven.) There are many orthogonal concepts we also need to understand, let alone the core business function of moving the money: How much money is being transferred? How much is the latency? How do I upgrade or evolve microservices? Can I scale it up and down? What happens to bad payments? Can it be extended for different payment types? Which teams are going to run my system? Where is it going to run?</p><p>For this reason, I’m not only going to focus on the payment processing pipeline but also work through the other functionality required to architect a production-worthy solution. By nature, anything that is event-driven provides loose coupling. With event-first design, the data becomes the API which, like any production system, needs to support change and evolution (i.e. <a href="https://avro.apache.org/">Avro</a> or <a href="https://developers.google.com/protocol-buffers/">Protobuf</a>). Furthermore, in embracing loose coupling, we inherit a clean <a href="https://en.wikipedia.org/wiki/Separation_of_concerns">separation of concerns (SoC)</a>, which provides the ability to develop and operate various aspects of the system architecture independently.</p><p>These aspects can be organized into a set of complementary pillars:</p><ol><li><strong>Business function:</strong> The payment processing pipeline</li><li><strong>Instrumentation plane:</strong> How much money is transferred? How quickly are payments processed?</li><li><strong>Control plane:</strong> Patterns to start, stop, pause, coordinate and autoscale</li><li><strong>Operational plane:</strong> Dead letter queues (DLQs), error tracking &amp; handling, logging and NoOps</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*hweeqq2quSzCsU-V.png" /><figcaption><em>The four pillars of event streaming architecture applied to our system, details excluded</em></figcaption></figure><h3>1. Business function</h3><p>The business function represents the core functionality being built. It provides business value that is immediate and beneficial to the organization. Examples include payment processing, logistics, data processing (ETL) and so on. This function can be developed as a set of events and related dataflows across one or more bounded contexts.</p><h3>2. Instrumentation plane</h3><p>The role of the instrumentation plane is to capture metrics that prove that the business function is performing sufficiently. Metrics may have a variety of uses-they might be used to drive alerts, assist in capacity planning or drive infrastructure automation functions such as auto-scaling.</p><p>In our case, business-level monitoring is the ability to track metrics such as <em>payments in progress</em>, <em>failed payments</em> and <em>payment throughput</em>. These could also be mapped onto application metrics, including the supporting dataflow volume, latency tracking, and throughput. In the next level down, they can be mapped to the underlying broker infrastructure metrics, such as consumer lag, throughput per topic and partition hotspots, in addition to operating system metrics like CPU, network I/O, disk I/O, load average, etc.</p><h3>3. Control plane</h3><p>An often overlooked aspect of many systems is its ability to control the flow of events. Unlike batch systems, event streaming applications continuously process data. This means that while it might be possible to perform rolling upgrades, the system sometimes needs to halt processing, perform an action, such as an upgrade or logic change, and then resume processing. The control plane becomes essential when outages have occurred and a large-scale system must get back online in a coordinated fashion, perhaps with incremental, restricted functionality.</p><p>In the context of a payment system, you might need to drain the current set of payments for a topology change. But what if there is a regional outage-how do you bring the system back online without overloading downstream systems?</p><h3>4. Operational plane</h3><p>Developing the system for the “happy path” is easy-it’s catering for the remaining paths when simplicity can become compromised. Relying on pushdown semantics of container “restart on failure,” workload feedback cycles to drive elasticity and other underlying functionality-we need to understand all of this and more. We need to build a model for capturing operational processes, such as rolling restarts and wipe and update processes, and develop these in automation scripts to drive the infrastructure automation.</p><p><a href="https://www.ansible.com/">Ansible</a> has emerged as a leading automation language and provides a wide range of plugins for <a href="https://www.ansible.com/use-cases/configuration-management">configuration management</a>, <a href="https://www.ansible.com/use-cases/orchestration">orchestration</a> as well as anything else that might be needed when working with physical servers. For those with a Kubernetes ecosystem, the use of <a href="https://helm.sh/">Helm</a>, <a href="https://helm.sh/docs/developing_charts/">Helm Charts</a> and <a href="https://github.com/operator-framework">Operator</a> scripts are quite popular.</p><p>The most important aspect of operations is not that you have dead letter queues or the like, but rather that you have trusted processes. These processes must be deterministic; they must be reliable. The only way of validating operational processes is to use automation and testing. Testing requires appropriate instrumentation that serves as a validation mechanism. Testing operational processes become a lot easier when the systems being operated are decoupled and, most importantly, have all its events captured in Kafka. This means that Kafka is used not only to store and emit events but also playback scenarios, so you can pinpoint when things started going wrong.</p><h3>Building the KPay payment system</h3><p><a href="https://github.com/confluentinc/demo-scene/tree/master/scalable-payment-processing">KPay</a> is a demonstration application that shows how to apply our event streaming pillars. It is a fully functioning payment system, providing a user interface (pictured below), a web tier, a series of <a href="https://docs.confluent.io/current/streams/index.html">Kafka Streams</a> processors as well as a payment generator for load testing. The payment generator and other parts of the web interface are exposed using Swagger (also pictured below). The source code is comprised of the following packages that map onto the pillars:</p><ul><li>payments: the business function of payment processing</li><li>metrics: the instrumentation plane to monitor payment throughput and latency</li><li>control: the control plane to stop and start payment processing</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*T6ErWYAVaGAan2hn.png" /></figure><p><em>The KPay user interface</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*DtRzd3JBVeysKtDN.png" /><figcaption><em>Swagger is used to document the REST interface and expose the control plane.</em></figcaption></figure><p>KPay leverages Kafka Streams and the various utilities it provides. These include materialized views called <a href="https://docs.confluent.io/current/streams/concepts.html#ktable">KTables</a>, where the collected state is natively backed by a Kafka topic called a <a href="https://docs.confluent.io/current/streams/concepts.html#duality-of-streams-and-tables">changelog stream</a>. Thus, Kafka Streams KTables also benefit from the scaling model provided by Kafka partitioning.</p><p>Kafka Streams processors are commonly called <a href="https://docs.confluent.io/current/streams/developer-guide/running-app.html">applications</a> because they are provided as a library and can be run within an application, or in a cluster using the technology of choice (Docker, naked Java processes, <a href="https://quarkus.io/guides/kafka-streams">Quarkus</a>, etc.). Another benefit of Kafka Streams is that a KTable will emit <a href="https://riccomini.name/kafka-change-data-capture-breaks-database-encapsulation">change-data-capture (CDC)</a> events through a connector (say <a href="https://www.confluent.io/hub">Elasticsearch</a>) to an external datastore, allowing it to act as a view layer. This mechanism separates the write path from the read path and is called <a href="https://www.confluent.io/blog/event-sourcing-cqrs-stream-processing-apache-kafka-whats-connection/">command-query separation (CQRS)</a>. Alternatively, we can query the KTables directly using <a href="https://docs.confluent.io/current/streams/developer-guide/interactive-queries.html">interactive queries</a>, also known as <a href="https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/">turning the database inside out</a>. I will discuss the second approach later on in this blog post.</p><h3>Pillar 1 — Business function: Payment processing pipeline</h3><p>Payment processing is an interesting problem. In addition to being deterministic, reliable, idempotent and scalable, more importantly, it can be user-driven or machine-driven. In either case, there can be challenges relating to latency and scale. Latency, where person X gets frustrated at the “busy spinner” and clicks a back button on a webform, or perhaps the purchase of a major sporting event ticket leads them to an error page- <em>did they take my money or not?</em> For synchronous machine-driven payments, client&#39;s latency can have a significant knock-on effect. It can force systematic retries or timeouts. If the client supports asynchronous interaction, be it machine or human, then the solution becomes simpler.</p><p><strong>Goal</strong></p><p><em>To handle millions of payments</em></p><p><strong>Requirements</strong></p><p>Although I normally write these as user stories, I’m just going to cut to the chase with a collection of fictional requirements. The system needs to report how much money is inflight, process both sides of the payment and also confirm the total amount of payments successfully processed. In other words, it must:</p><ul><li>Scale to millions of payments</li><li>Perform asynchronous processing</li><li>Track payments inflight (total value of payments being processed)</li><li>Remove/debit money from an account when handling a debit event</li><li>Add/credit money to an account when handling a credit event</li><li>Track payments confirmed today (total value of payments successfully processed)</li></ul><p><strong>Event flow model</strong></p><p>The payment processing pipeline is an event streaming dataflow that uses a series of Kafka Streams applications for each stage. Each stage performs a specific function, tracks a payment, update a balance, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*25cY9di7SnBXTAXi.png" /><figcaption><em>Payment processing pipeline</em></figcaption></figure><p>Let’s walk through the flow of the above diagram:</p><ol><li>Payment requests flow from the payment.inflight topic into the payments inflight processor. It updates the total balance of payments currently being processed, similar to a read-uncommitted view. It sends the payment request into the payment.inflight topic, keyed by the from account and setting the event type to DEBIT.</li><li>An account processor receives the DEBIT request, validates that the payment can be processed and updates the user&#39;s balance (KTable). It then emits another event, keyed against the to account, and sets the event type to CREDIT.</li><li>Another account processor receives the CREDIT event, updating the user&#39;s balance in the KTable and emitting the payment-complete event.</li><li>The payments conf’d processor updates the total confirmed balance, which is similar to the read-committed value.</li></ol><p>The account processors (from 2 and 3) handle both sides of the payment (debit and credit). Depending on the event.key, Bob and Alice may or may not be handled by the same instance.</p><h3><strong>Event flow breakdown</strong></h3><h4>1. Payments inflight</h4><p>Payments inflight acts as a gateway for the business flow. Its responsibility is to track business-level metrics that are important to application users, including the total payment value inflight and the number of payments currently being processed.In contrast, instrumentation metrics relate to the application support team and might include latency percentiles, payment throughput and outliers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*o4ObtlCtrCoGu0bP.png" /></figure><p>The payment is received on payment.incoming, the total payment amount inflight, and an output event is emitted onto payment.inflight. When a payment was received from payment.complete and has a status of <a href="https://github.com/confluentinc/demo-scene/tree/master/scalable-payment-processing/src/main/java/io/confluent/kpay/payments/model/Payment.java">payment.state.complete</a>, then the payment value is removed.</p><p>The code below shows the payments inflight processor and the inflights aggregator class that maintains the state. It updates the account balance according to debit or credit payment.state. Notice how the processor has two parts: one for the statistical aggregation and another to rekey and emit the payment event to the payment.inflight topic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*RC0N52-0PD8_2FHL.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*Q_Z9R7dsxHO4yixz.png" /></figure><h4>2–3. Account Processor</h4><p>The account processor maintains the value of a user’s account by handling debit and credit payments. Depending upon the type of payment being processed, a subsequent payment event will be emitted to the payment.inflight or payment.complete topic. Interactive queries display the AccountBalance values through the user interface.</p><p>In this case, the code uses a generic set of classes to <a href="https://github.com/confluentinc/demo-scene/tree/master/scalable-payment-processing/src/main/java/io/confluent/kpay/payments/AccountProcessor.java#L96">expose the information via REST</a>. The alternative approach is to use <a href="https://martinfowler.com/bliki/CQRS.html">CQRS</a>, where the KTable’s changelog event stream is used to project account balances into a viewing service like Elastic, Cassandra or something similar. Note: The default behavior of a KTable is to emit changelog events.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*ztirwbIp1-QxG-1x.png" /></figure><p>The code below shows the AccountProcessor followed by the AccountBalance aggregator. Of the two branches within this topology, one maintains state via the AccountBalance aggregator (.groupByKey), and the other routes payment based upon the State ( isCreditRecord or isCompleteRecord) .map.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*oercnRgBo1B1dowU.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*izkiX5LVHyp_MFlY.png" /></figure><h4>4. Balance confirmed</h4><p>The final stage is <em>confirmation</em>. Payments emitted from this stage have been successfully processed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*AbzG7wHquBtqKwzQ.png" /></figure><p><strong>Scaling model and limitations</strong></p><p>Notice how the above code reduces the key space of the groupBy(key). Rekeying using values 0-9 will distribute the payments to an intermediate topic across a limited set of partitions (the topic should be configured with 10 partitions). The purpose is to allow multiple consumers to aggregate their allocated set of partition data. If there is a single Kafka Streams processor, it will receive all partitions; however, as more are added, the data will balance across them according to the <a href="https://docs.confluent.io/current/clients/consumer.html">consumer group protocol</a>. Aggregation is performed on a per-key basis, so there will only be 10 <a href="https://github.com/confluentinc/demo-scene/tree/master/scalable-payment-processing/src/main/java/io/confluent/kpay/payments/model/ConfirmedStats.java">ConfirmedStats</a> instances.</p><p><strong>The data serving layer: Interactive queries and materialized views</strong></p><p>The ability to maintain real-time data across a series of stateful, stream processing microservices is very powerful, but how do we make that state accessible? How do we see how much money is in each person’s account and serve data to interested clients? In order to access stateful data, you can either <a href="https://www.confluent.io/blog/event-sourcing-cqrs-stream-processing-apache-kafka-whats-connection/">use CQRS by emitting CDC events</a> to an <a href="https://www.confluent.io/hub">external datastore</a>, such as <a href="https://www.confluent.io/hub">Elastic</a>, or query the KTable directly. I will focus on the latter.</p><p>Kafka Streams supports materialized views similar to a relational table, albeit with a key-value (KV), iterator-style API. This model is called <a href="https://docs.confluent.io/current/streams/developer-guide/interactive-queries.html">interactive queries</a>, or queryable state/tables, and it provides just enough scaffolding for the user to leverage their preferable networking layer, whether it’s <a href="https://grpc.io/">gRPC</a> or <a href="https://en.wikipedia.org/wiki/Representational_state_transfer">REST</a>. <a href="https://github.com/confluentinc/demo-scene/tree/master/scalable-payment-processing">KPay</a> uses interactive queries extensively to provide state for the rendering of account balances, business metrics and instrumentation metrics, among other activities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*E-CMpQH5b_HRK90z.png" /></figure><p><em>KTable state (live account balances per user in this example) is exposed to the web tier using interactive queries.</em></p><h4>Kafka Streams interactive queries scaffolding</h4><p>Kafka Streams provides an endpoint-agnostic means to building your own interactive queries layer. This supports real-time discovery of endpoints as processors scale up or down.</p><p>The approach is straightforward, relying on a <a href="https://docs.confluent.io/current/streams/concepts.html#globalktable">GlobalKTable</a> to share application endpoint information between instances of the same application ID. Like regular KTables, they are backed by a changelog topic. When a KTable is created, the <em>store name </em>and<em> endpoint address</em> are specified and published as <a href="https://docs.confluent.io/current/streams/javadocs/org/apache/kafka/streams/state/StreamsMetadata.html">StreamsMetadata</a>. The endpoint information is available to all remote Kafka Streams contexts using the <a href="https://github.com/confluentinc/demo-scene/tree/master/scalable-payment-processing/src/main/java/io/confluent/kpay/rest_iq/MetadataService.java">MetadataService</a> that queries the <a href="https://docs.confluent.io/current/streams/javadocs/org/apache/kafka/streams/KafkaStreams.html#allMetadata--">KStreams.allMetatadata()</a>.</p><p><strong>KPay interactive queries: KTables</strong></p><p>The REST interactive queries layer that I’ve developed is extremely simple and reusable. It relies on class introspection to minimize the amount of boilerplate code required. Feel free to test drive the simplified <a href="https://github.com/confluentinc/demo-scene/tree/master/scalable-payment-processing/src/test/java/io/confluent/kpay/rest_iq">tests</a> and then dive into the <a href="https://github.com/confluentinc/demo-scene/tree/master/scalable-payment-processing/src/main/java/io/confluent/kpay/rest_iq">code</a>.</p><p>To build a discoverable REST endpoint for a KTable or WindowedKTable, implement code similar to the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*f4oDJ5NFrVM0aIiQ.png" /></figure><p>The <a href="https://github.com/confluentinc/demo-scene/tree/master/scalable-payment-processing/src/main/java/io/confluent/kpay/rest_iq/WindowKTableResourceEndpoint.java">WindowKTableResourceEndpoint</a> class introspects the provided KTable&#39;s generic parameters to construct the REST endpoint, which is then available to drive via <a href="https://swagger.io/">Swagger</a>. The client-side of the REST service uses these classes within the <a href="https://github.com/confluentinc/demo-scene/tree/master/scalable-payment-processing/src/main/java/io/confluent/kpay/rest_iq/KTableRestClient.java">KTableRestClient</a> to discover and access remote instances.</p><p>The following code is used to create a client that queries each remote <a href="https://github.com/confluentinc/demo-scene/blob/master/scalable-payment-processing/src/main/java/io/confluent/kpay/payments/AccountProcessor.java">AccountProcessor</a> endpoint to obtain the live balance of user accounts. It shows how to request the full set of account keys as a Set&lt;String&gt; and then retrieve the set of account balances.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*iwLZnJbH-xsLb3Ck.png" /></figure><p><em>The </em><em>KTableRestClient lists all </em><em>accountKeys and then fetches </em><em>accountBalances.</em> <strong>Enhancements</strong></p><p>The purpose of the demo is to demonstrate the four key architectural principles in practice. It can be improved with regard to:</p><ul><li><strong>Error handling:</strong> Implement <a href="https://microservices.io/patterns/data/saga.html">sagas</a> to handle unwinding on failed payments, perhaps when corrupted or fraudulent activity is detected</li><li><strong>Pre-payment validation:</strong> Add a preliminary stage to support scenarios such as detecting and handling suspicious or fraudulent activity</li><li><strong>Restrict dependent payments:</strong> When it makes sense, only allow a single debit payment for a single account to be inflight at any point in time</li><li><strong>Limits:</strong> Introduce limit checks against accounts with low balances</li></ul><h3>Pillar 2 — Instrumentation plane: Business metrics</h3><p>There are a plethora of monitoring tools that provide valuable functionality. Most of them are great for bottom-up metrics or infrastructure metrics.</p><p><strong>Operating system and application infrastructure metrics are useful for overall knowledge, but the most important question to answer is: Am I meeting business goals?</strong></p><p>The goal of the instrumentation plane is tracking business metrics. We need to know with certainty that the system is performing correctly and how system performance impacts the business. <em>Are our users happy? Are we responding quickly enough to meet demand?</em> Behavioral problems can often be identified through these business metrics because they tend to result from a repeated series of underlying interactions that comprise the function, the flow of events. Another aspect is that business metrics can be used to drive adaptive behavior. Examples include scaling up the processing capability, applying backpressure and modifying system SLAs.</p><p><strong>Goal</strong></p><p><em>To meet the system’s business goals</em></p><p><strong>Requirements</strong></p><p>Let’s assume we have the following requirements, some of which should be quite obvious:</p><ul><li>Process payments in &lt; 200 milliseconds (latency)</li><li>Throughput &gt; 1,000 per second</li><li>Report the total count and dollar amount (the demo code uses a one-minute window)</li></ul><p><strong>Event flow model</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*jBVgQBeTUNpe8beY.png" /></figure><p>The instrumentation flow model is a single stream processor that uses the timestamp information of each payment to drive an aggregator. Latency is measured by the time-delta between the current time and the timestamp set by the payments inflight processor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*GH-a9WG2E4nttUvs.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*HLe6-bk4g-R4kb3L.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*_UfFqNmxoX2hfcT_.png" /></figure><p><em>The user interface rendering throughput statistics</em></p><h3>Pillar 3 — Control plane: Flow control, start, pause, bootstrap, scale and rate limit</h3><p>The last pillar of the event streaming architecture is the control plane, which drives and coordinates the system. The demo allows you to halt and resume the processing of payments, in which the former is particularly valuable because it allows the pipeline to complete inflight processing. This essentially provides a flush (or drain) of the current set of inflight payments, which can be useful when there is a topology or logic change.</p><p>It should be noted that when changing an event stream’s topology, the intermediate topics may have a different structure. It is not always a straightforward upgrade path. This kind of functionality can also be leveraged to redirect flow onto a new set of processors, or drive A/B processing or an upgrade path. For example, when a topology change occurs, the current pipeline is halted and flushed, state stores are backed up and state is migrated to new instances that take over the processing. The new instances prepopulate the backed-up state, and then processing is resumed.</p><p><strong>Goal</strong></p><p><em>To pause and resume payment processing</em></p><p><strong>Requirements</strong></p><ul><li>Pause processing and allow existing payments to be completed (drained)</li><li>Resume processing from the paused state</li></ul><p><strong>Event flow model</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*8NPZAqiIKGmuH-eI.png" /></figure><p>The StartStopController shown below handles start/pause events that drive a callback to pause() and resume() methods. The methods call back onto a <a href="https://github.com/confluentinc/demo-scene/blob/master/scalable-payment-processing/src/main/java/io/confluent/kpay/control/Controllable.java">Controllable</a> instance injected at creation time. The basic implementation uses object.wait and notify methods as seen in <a href="https://github.com/confluentinc/demo-scene/blob/master/scalable-payment-processing/src/main/java/io/confluent/kpay/control/PauseControllable.java">PauseControllable</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*0Ny3nkXemFeH6OZ2.png" /></figure><p>The PauseControllable passes into the entry point for the payment processing pipeline, the InflightProcessor. It then interacts with the event stream&#39;s .filter() call as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*plTkH62f8BWxIQnU.png" /></figure><p><a href="https://github.com/confluentinc/demo-scene/blob/master/scalable-payment-processing/src/main/java/io/confluent/kpay/payments/PaymentsInFlight.java#L53"><em>PaymentsInflight.java</em></a><em> uses a </em><em>filter() to pause processing.</em></p><p>This particular approach is abstracted away via the controller interface. The downside is that we have introduced interface coupling between the business and control plane. An alternative approach is to introduce a gated processor that controls the flow of events into another topic. The downside of the gated approach, however, is that it incurs another Kafka topic.</p><h3>Pillar 4 — Operational plane: Event logging, DLQs and automation</h3><p>Operations aren’t a new concept; however, the new age of NoOps, configuration management, and automation, etc., encompasses operational concerns under the banner of DevOps. I will provide a view operational dataflows that can be used to support this pillar. To state the obvious, the operational plane is entirely dependent on the instrumentation plane for observability and the control plane for coordination. Operational aspects include, upgrade processes, 24×7 runtime support, evolutionary support (e.g., new processors or data evolution) and, of course, the infrastructure to handle normal operations. The dataflow patterns normally introduced are summarized below:</p><ul><li><strong>Application logs:</strong> Each microservice will use a log appender to selectively send log messages to a specific topic that is relevant to the application context. The log message will also contain metadata about the runtime, perhaps including the input topic and output topic, consumer group as well as the source event information. These logs are relevant to business events and should be tied to the bounded context.</li><li><strong>Error/warning logs:</strong> Similar to above, but the focus is on error/warning log events and sending to the relevant topic. It is also useful to build a log context that accumulates relevant information to enrich the log message if an error/warning does occur.</li><li><strong>Audit logs:</strong> Each microservice will capture a security context (e.g., user, group, credential or reason) as well as event context (i.e., event ID, event origin, source and destination topic) and emit to a relevant, contextually determined audit log topic.</li><li><strong>Lineage:</strong> Similar to above, except that the event will contain trace information about its journey-for example, each processor, or the context it has accessed or modified on its dataflow.</li><li><strong>Dead letter queues:</strong> These provide a place to store raw events that cannot be processed. Processing failure can be due to any number of reasons, from deserialization failure to invalid values or invalid references (join failure, etc). In any case, the purpose of dead-letter queues is to make you aware of the error so you can fix it. The dead letter queue is the last line of defense and might indicate a failing CI/CD process, in which a scenario has occurred that was not catered for. The dead letter queue should always be empty. In some cases, a <a href="https://microservices.io/patterns/data/saga.html">saga</a> pattern is needed to unwind the preceding state that caused the error. Remember, events must be processed idempotently.</li></ul><p>These patterns all need to be associated with the bounded context in which they occur.</p><h3>Putting it all together</h3><p>The quick and lazy version of running the demo is within your IDE (captured below).To run it in a more robust manner, you can Dockerize the relevant components. Each stream processor and the web tier contain their own main() method. The main() method can be used to run the appropriate class within the Docker image. To see how all stream processing microservices run within a monolith, refer to <a href="https://github.com/confluentinc/demo-scene/blob/master/scalable-payment-processing/src/main/java/io/confluent/kpay/KPayAllInOneImpl.java">KPayAllInOneImpl</a>.</p><p>Run the payment system within your IDE:</p><ol><li>git clone git@github.com:confluentinc/demo-scene.git</li><li>Run IntelliJ, then open ./scalable-payment-processing/ and import the maven.pom file</li><li>Run/debug the RestEndpointIntegrationTest.runServerForABit()</li><li>Point the browser to <a href="http://localhost:8080/ui/index.html#">http://localhost:8080/ui/index.html#</a></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*GVtOCIL_OsuZMRKg.png" /></figure><p>5. The <a href="https://swagger.io/">Swagger</a> interface is available at http://localhost:8080/swagger/index.html. It exposes the REST API and JSON data model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*7KPlNHv5NUQXZr_f.png" /></figure><h3>Deployment model</h3><p>If you choose to run the payment system within Kubernetes and Docker, each plane ends up running separately. In the deployment below, four account processors are running. The web tier queries each AccountProcessor using the <a href="https://github.com/confluentinc/demo-scene/tree/master/scalable-payment-processing/src/main/java/io/confluent/kpay/rest_iq/MetadataService.java">stream.metadata</a> to interact with each instance as a KV store and <a href="https://github.com/confluentinc/demo-scene/tree/master/scalable-payment-processing/src/main/java/io/confluent/kpay/KPayAllInOneImpl.java#L215">returns the JSON data</a> for rendering in the browser.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*S0XUOBiyIjeO-k9q.png" /></figure><p>There are, of course, many shortcuts being made in the <a href="https://github.com/confluentinc/demo-scene/tree/master/scalable-payment-processing/src">codebase</a>. Here are the most obvious ones:</p><ul><li>Event objects are exposed through the web tier. This is bad because they need to be contained within the bounded context. There should be a translation layer into POJOs to support evolvability that is independent between the web clients and the bounded context. In short, don’t bleed the domain model (events) outside of a bounded context.</li><li>There is virtually no error handling, though we need dead letter queues, error queues and saga implementation and handling.</li><li>There is no configuration or environmental management to support automation, which is required for topic configuration, broker connectivity, security and so on.</li><li>There is a lack of versioning.</li></ul><p>In the near future, we will be merging KPay into the existing <a href="https://github.com/confluentinc/kafka-streams-examples/tree/5.0.0-post/src/main/java/io/confluent/examples/streams/microservices">streaming microservice demo ecosystem</a>.</p><h3>Parting thoughts and preparing for the future</h3><blockquote>The “software crisis” was first identified in 1968 [NR69, p70] and in the intervening decades has deepened rather than abated. <br><a href="https://github.com/papers-we-love/papers-we-love/blob/master/design/out-of-the-tar-pit.pdf">Out of the Tar Pit, 2006</a></blockquote><p>I started this post by talking about complexity. Complexity in system design is inevitable in any serious, large-scale system. To tackle complexity, you need control. To have control, you need to observe. To observe, you need to instrument. While control, observability, and instrumentation are commonplace in many traditional systems, their implementation is subtly different in the world of events. We demonstrated this by reimagining a traditional payment system as a set of discrete stream processors. We saw how, despite being stateful, such a system could be scaled to accommodate digital-native workloads. Moreover, we saw how to layer control and observability over systems of this type, identifying four key pillars in the process-pillars that not only apply to payment processing but also to any event streaming system we may choose to build. These pillars are:</p><ol><li><strong>Core business flow:</strong> business functionality</li><li><strong>Instrumentation plane:</strong> metrics and observability</li><li><strong>Control plane:</strong> start, stop and scale</li><li><strong>Operations plane:</strong> application support, error handling, and dead letter queues</li></ol><p>When the composition approach is applied across the organization, we start getting into the realms of events as a backbone and a governance model and more. To support all of it, we also need a strong <a href="https://theagileadmin.com/what-is-devops/">DevOps story</a> (<a href="https://medium.com/@neil.avery_68603/tools-for-event-driven-devops-with-apache-kafka-f4ca1721cc2a">Kafka DevOps tools</a>).</p><p>The natural next step in this journey is for these pillars to push into the language and runtime environment. Instrumentation, control planes and elasticity become part of the language runtime which, at deploy time, figures out: where the dataflow is going to run, which cloud, container or FaaS. The choreographed dataflow that captures the bounded context becomes how you think about business functions connected through orchestration.</p><p>It feels like the last 30 years of distributed systems have left a trail of unloved technology scattered down a dirt track, and only now we’ve started building distributed systems the right way. Through this series, I wanted to explain why you should build applications using event streaming, and why it’s the right approach when others have failed.</p><p><a href="https://www.confluent.io/blog/journey-to-event-driven-part-1-why-event-first-thinking-changes-everything">Part 1</a> highlighted the value of questioning the status quo, hence the title. <a href="https://www.confluent.io/blog/journey-to-event-driven-part-2-programming-models-event-driven-architecture">Part 2</a> reasoned about why using an event streaming platform provides everything you need to build large-scale, event streaming and stateful applications, without most of the “microservicey” legacy frameworks. The goal is essentially to keep it simple and not boil the ocean. In <a href="https://medium.com/@neil.avery_68603/journey-to-event-driven-part-3-the-affinity-between-events-streams-and-serverless-apache-84bd37d0f9b9">part 3</a>, I really felt like FaaS or serverless functions needed a place to exist, and showed how they are complementary to the event streaming application. And finally, here in part 4, we connected all of the dots and demonstrated how to think about the architecture, using composition and patterns to control complexity in order to build applications at any scale.</p><p>I hope you have enjoyed the journey so far. Stay tuned while we continue to stand on the shoulders of giants.</p><h3>Interested in learning more?</h3><p>Head over to <a href="http://liquidlabs.com/">liquidlabs.com</a> and see what services we offer, as well as other blog posts about technology (not just Kafka, but serverless opensource projects).</p><p>Otherwise, if you’d like to know more, you can <a href="https://www.confluent.io/download/">download the Confluent Platform</a>, the leading distribution of Apache Kafka.</p><p>To read the other articles in this series, see:</p><ul><li><a href="https://medium.com/@neil.avery_68603/journey-to-event-driven-part-1-why-event-first-thinking-changes-everything-c98b98e248d5">Journey to Event-Driven — Part 1: Why Event-First Thinking Changes Everything</a></li><li><a href="https://medium.com/@neil.avery_68603/journey-to-event-driven-part-2-programming-models-for-the-event-driven-architecture-eded219630a2">Journey to Event Driven — Part 2: Programming Models for the Event-Driven Architecture</a></li><li><a href="https://medium.com/@neil.avery_68603/journey-to-event-driven-part-3-the-affinity-between-events-streams-and-serverless-apache-84bd37d0f9b9">Journey to Event-Driven — Part 3: The Affinity Between Events, Streams and Serverless</a></li><li><a href="https://medium.com/@neil.avery_68603/journey-to-event-driven-part-4-four-pillars-of-event-streaming-microservices-apache-kafka-9e88fd520a07">Journey to Event Driven — Part 4: Four Pillars of Event Streaming Microservices</a></li></ul><p>As a technologist at Liquidlabs, Neil Avery is an industry expert in the world of streaming, distributed systems and the next generation of technology. Various aspects of the function include working with prominent customers, working with innovators to drive innovation into the Kafka ecosystem and thought-lead about the next frontier of innovation. He has over 25 years of expertise of working on distributed computing, messaging and stream processing. He has built or redesigned commercial messaging platforms. You can also follow him on <a href="https://twitter.com/avery_neil">Twitter</a>.</p><p><em>Originally published at </em><a href="https://www.confluent.io/blog/journey-to-event-driven-part-4-four-pillars-of-event-streaming-microservices"><em>https://www.confluent.io</em></a><em> on May 9, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e88fd520a07" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Journey to Event-Driven — Part 3: The Affinity Between Events, Streams and Serverless | Apache…]]></title>
            <link>https://medium.com/@neil.avery_99/journey-to-event-driven-part-3-the-affinity-between-events-streams-and-serverless-apache-84bd37d0f9b9?source=rss-68d2da6ed2c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/84bd37d0f9b9</guid>
            <category><![CDATA[serverless-architecture]]></category>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[event-driven-systems]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[event-driven-architecture]]></category>
            <dc:creator><![CDATA[Neil Avery]]></dc:creator>
            <pubDate>Wed, 27 Feb 2019 17:45:50 GMT</pubDate>
            <atom:updated>2020-02-12T11:36:08.639Z</atom:updated>
            <content:encoded><![CDATA[<h3>Journey to Event-Driven — Part 3: The Affinity Between Events, Streams and Serverless | Apache Kafka</h3><p><em>Please note, this version has been updated since it was originally published on the Confluent blog; many things have changed! (</em><a href="https://www.confluent.io/blog/journey-to-event-driven-part-3-affinity-between-events-streams-serverless/"><em>link</em></a><em>)</em></p><p>With serverless being all the rage, it brings with it a tidal change of innovation. Given that it is at a relatively early stage, developers are still trying to grok the best approach for each cloud vendor and often face the following question: Should I go cloud-native with AWS Lambda, GCP functions, etc, or invest in a vendor-agnostic layer like the <a href="https://serverless.com/">serverless framework</a>?</p><p>Developers also need to make choices on the dark art of state management. FaaS functions only solve the compute part, but where is data stored and managed, and how is it accessed? This is important as each platform has different persistence models and APIs, tying you to that vendor’s ecosystem. What is more, as the world adopts the event-driven, streaming architecture, how does it fit with serverless? Do they complement or compete? When it comes to the emerging serverless world, It makes sense to validate how Apache Kafka ®fits in considering that it is <a href="https://www.confluent.io/apache-kafka-report/">mission-critical in 90 percent of companies</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*eXz48X2h8sCPH8m1.png" /></figure><blockquote>Serverless functions provide a synergistic relationship with event streaming applications; they behave differently with respect to streaming workloads but are both event driven</blockquote><p>In <a href="https://medium.com/@neil.avery_68603/journey-to-event-driven-part-1-why-event-first-thinking-changes-everything-c98b98e248d5">part 1</a> of this series, we developed an understanding of event-driven architectures and determined that the event-first approach allows us to model the domain in addition to building decoupled, scalable and enterprise-wide systems that can evolve.</p><p>The key to event-first systems design is understanding that a series of events captures behavior. By persisting the streams in Kafka we then have a record of all system activity (a source of truth), and also a mechanism to drive reactions. Normally these events a handled by stream processors or consumers; however, in this case, we want to understand how FaaS functions fit into the event streaming model.</p><p>We will also explore how FaaS runtime characteristics make it suitable for different types of processing, as in some cases, latency or concurrency concerns need to be catered for. After working through the requirements of stream processing, I’ll then develop several principles showing that FaaS does indeed work with the model of event streams, despite some caveats.</p><h3>Overview</h3><ul><li>What is FaaS?</li><li>Event-first FaaS?</li><li>FaaS for streaming processing</li><li>FaaS as part of the event-driven, streaming architecture</li><li>What’s next?</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*-4Mi-zPbTfsEGd-6.png" /><figcaption>What is FaaS?</figcaption></figure><p>The thing I like most about FaaS is the name and how everyone uses it as an attempt at humor. I will resist ;).</p><p>FaaS is the ability to take a function and run it somewhere in the cloud; it’s the “compute” part of the serverless stack where you bring your own code. The function contains a bespoke logic block. It is then called via some kind of registry like an API gateway, or it is scheduled or triggered by a cloud-related event (i.e., data written to Amazon S3).</p><p>The function is expected to run for a period of time and then exit, with cloud vendors charging by the millisecond (and associated memory). The functions can have many instances running in parallel, and after the first “cold” start call, they are considered “hot.” The first call is where database connections (and the like) should be initialized. When calling via an API gateway, functions might be called synchronously to return a value.</p><p>There are many benefits to FaaS:</p><ul><li>Lightweight</li><li>Decoupled</li><li>Scalable</li><li>Very cost-efficient (pay per use)</li></ul><p>Many people predict that we will just be building cloud apps via the composition of FaaS functions. They moved from simple use cases like, “build a thumbnail of this image,” to mainstream application logic like process payments.</p><blockquote><em>“The management of uncertainty must be implemented in the business logic.” </em><a href="http://adrianmarriott.net/logosroot/papers/LifeBeyondTxns.pdf"><em>Pat Helland</em></a></blockquote><p>The 2 primary challenges (and hot topics) with Serverless functions are:</p><ol><li>State management. Being stateless and unreliable means anything requiring memento has to cope with unreliability, and/or recovering state. There are a couple of proposals: <a href="https://blog.acolyer.org/2020/02/07/cloudburst/">cloud-burst</a>; which seems like a hack on distributed caching/events — ala Hazelcast/Coherence and <a href="https://cloudstate.io/">CloudState</a> — which points in a similar direction, by owning the invocation path; it&#39;s driven by Akka.</li><li><a href="https://www.slideshare.net/NeilAvery1/serverless-london-2019-faas-composition-using-kafka-and-cloudevents?ref=https://serverlesscomputing.london/sessions/faas-composition-using-kafka-and-cloud-events/">Composition</a>/Workflow. <a href="https://aws.amazon.com/step-functions/">AWS Step Functions</a> and <a href="https://docs.microsoft.com/en-us/azure/azure-functions/durable-functions-overview">Azure Durable Functions</a> (patterns) are proposed by the relevant vendors. I presented the concept of using stream processing — map and <a href="https://www.slideshare.net/NeilAvery1/serverless-london-2019-faas-composition-using-kafka-and-cloudevents?ref=https://serverlesscomputing.london/sessions/faas-composition-using-kafka-and-cloud-events/">flatmap</a> at this <a href="https://serverlesscomputing.london/sessions/faas-composition-using-kafka-and-cloud-events/">conference</a>.</li></ol><h3>Event-first FaaS</h3><p>Applying event-first principles to FaaS is relatively straightforward. From an infrastructure perspective, you need a streaming provider through which the events are stored and used to trigger FaaS functions. With Kafka, this would involve using a connector to bridge between a Kafka topic and the FaaS function. As each event is received, it calls on the specified FaaS function.</p><p>All of the above principles hold true from an architecture perspective as well. The event models a fact, the FaaS function reacts to that fact; it might generate an event of its own. The nice thing about this is that it moves away from the FaaS integration challenge. It is common to use FaaS to react to events from S3 or other parts of your cloud infrastructure. The challenge is the inherent scale and complexity as your application grows, coupled with the loss of ability to trace events and manage functionality. By routing all events through streams, they are stored as facts. These facts are consumed by the <a href="https://www.confluent.io/hub">connector</a> which is then used to trigger the appropriate FaaS function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/0*B2qKprxkZeoYVG_S.png" /></figure><h3>FaaS for stream processing</h3><p>In our event-driven, streaming application, we have the behavior of our domain model being captured (persisted) as events within streams. Stream processors allow us to work natively with these streams in a “correct” fashion by supporting a myriad of patterns with either bespoke logic, such as Kafka Streams, or a higher-order grammar: KSQL. These are composed as an event stream that works quite well…which leads us to the question: When and where should FaaS be used? Can I use it to enrich user information from external sources or filter against a particular set of users?</p><p>To answer this accurately, we need to establish the characteristics that make FaaS well suited to different types of processing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/1*9j6S028Emurefa1I4oGJjQ.png" /></figure><p>As conveyed through the quote below, even after eliminating the common considerations of FaaS, there may still be adverse side effects and runtime characteristics due to implementation side effects, including function contention, resource governance, etc.</p><blockquote>We characterize performance in terms of scalability, cold start latency, and resource efficiency, with highlights including that AWS Lambda adopts a bin-packing-like strategy to maximize VM memory utilization, that severe contention between functions can arise in AWS and Azure, and that Google had bugs that allow customers to use resources for free. <br><a href="http://pages.cs.wisc.edu/~liangw/pub/atc18-final298.pdf"><em>Peeking Behind the Curtains of Serverless Platforms, 2018</em></a></blockquote><p>Mapping these together, we begin to see the overlap where FaaS affords a cost-efficiency currently unavailable with stream processing.</p><ul><li><strong>Low latency:</strong> Single event processing in FaaS is too slow for stream processing (<a href="http://pages.cs.wisc.edu/~liangw/pub/atc18-final298.pdf">for a warm start, AWS: 25 ms, Google: 79 ms and Azure: 320 ms</a>). However, record batching into groups of events overcomes latency concerns. Note: Async <a href="https://docs.aws.amazon.com/lambda/latest/dg/limits.html">payload limit</a> = 128 KB. Sync <a href="https://docs.aws.amazon.com/lambda/latest/dg/limits.html">payload limit</a> = 6 MB. <strong>YES</strong></li><li><strong>High throughput:</strong> By batching records together and utilizing parallelism, significant throughput can be achieved, i.e., five parallel functions processing 1,000 events every 200 ms provides a throughput of 5 x 1,000 per 200 ms = 25,000 events per second. <strong>YES</strong></li><li><strong>Stateless operation:</strong> Filtering, transformation, and enrichment (i.e., load from a static table) are possible. <strong>YES</strong></li><li><strong>Stream-oriented stateful operations:</strong> Stream-stream/stream-table joins, windowing, etc., aren’t natively supported. <strong>NO</strong></li><li><strong>Non-streaming stateful operations:</strong> Stream-external-table joins and stream enrichment from external sources are possible provided the external data is not in motion. <strong>YES</strong></li><li><strong>Stream correctness (order preservation):</strong> There are no guarantees about invocation based container reuse, meaning correctness cannot be guaranteed beyond the scope of a single batch invocation. <strong>NO</strong></li><li><strong>Stream patterns:</strong> Patterns are stateful-they are not possible.<strong> NO</strong></li></ul><p>Amazon’s recently published a white paper <a href="https://d1.awsstatic.com/whitepapers/Serverless_Streaming_Architecture_Best_Practices.pdf">Serverless Streaming Architectures and Best Practices</a> is a great read and makes some good points that should be mapped onto the constraints above.</p><blockquote>AWS Lambda and serverless architectures are well-suited for stream processing workloads which are often event-driven and have spiky or variable compute requirements. <br><a href="https://d1.awsstatic.com/whitepapers/Serverless_Streaming_Architecture_Best_Practices.pdf"><em>Serverless Streaming Architectures and Best Practices, 2018</em></a></blockquote><p>As shown in the recipes section of the paper, FaaS is great where events are treated atomically and are (mostly) stateless. They are processed in isolation to perform simple enrichment and filtering before passing to a storage layer or queue. If the system needs to guarantee event ordering when writing to an output stream then, as previously discussed, concurrent FaaS execution will lead to corruption unless they are synchronously driven at the partition level.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/0*F5DRDYkgT-gjL3fT.png" /><figcaption><em>A source stream running through concurrent FaaS invocations will break event ordering.</em></figcaption></figure><p>For non-stream-oriented stateful processing, such as those that enrich against (static) external resources, then FaaS makes a lot of sense provided that the operation is not time-sensitive. For example, stream-table join operations to enrich a user ID to username and address are easily supported where the event stream contains the user ID, and where DynamoDB contains a user table.</p><p>If the user information is updated then subsequent invocations against hot state could be incorrect. If stale data cannot be tolerated then the FaaS instance would need to check and reload for every event being processed.</p><p>The main problem with this approach (for non-static data) is that it is not event-driven; you should never “fetch data.” Where time sensitivity is essential, the data should be provided as part of the event and processed using a stream processor that is time-aware as part of its runtime (i.e., join events within a time window). Secondly, the cost of the “fetch” drives up the cost of the FaaS. You could pay to wait for an RPC which introduces latency; the alternative is to cached data locally, but then how would you know it was stale or how long to cache it for?</p><p>The final performance consideration is latency. FaaS is problematic with cold starts (three seconds for Java processes), and hot invocations are in the vicinity of 200 ms-300 ms. Resorting to batching overcomes some of the latency, but there is still an initial hit that results in erratic performance.</p><blockquote><strong>Caveat emptor:</strong> FaaS gives us an excellent solution where processing is atomic (stateless), reliable latency is not a concern (1–100 ms) and the order of processing is not important. Stateful processing is also suitable provided that it is against an external resource and stale data concerns are understood.</blockquote><p>In the context of the streaming dataflow model, we can state the following.</p><p><strong>FaaS event-driven principles:</strong></p><ol><li><strong>In band but edge</strong> (stateless on the way in or out), i.e., map user GeoIP to a geocell</li><li><strong>In band, stateless and not latency sensitive</strong></li><li><strong>In band and enriched against external resources</strong>, i.e., enrich a user’s address</li><li><strong>Out of band, but edge:</strong> This is FaaS processing on a known set of data in response to an event where there is no downstream stream processing. For example, you can perform large-scale analytic processing of all auction site bidders against “cars in 2018” (a non-event-streaming problem).</li><li><strong>Ad hoc requests, but not streaming:</strong> They tend to be historic. If a large set of data is to be processed, then it is likely to be batch-oriented. Historic analytics include Monte Carlo simulation, raw number-crunching of event data, etc.</li></ol><h3>FaaS as part of the event-driven, streaming architecture</h3><p>Fundamentally speaking, you can leverage AWS Lambda integration with Amazon Kinesis or the Kafka Connect AWS Lambda Sink Connector to trigger events from streams. So it depends on the orientation of your development methodology. Do you want to stick to the cloud vendor platform or adopt a vendor-agnostic approach? Many organizations favor the latter, which is the approach I will use.</p><p>In this example we are considering an auction platform:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/0*jUmzsjcACTlHmN9D.png" /><figcaption><em>The event stream of an auction system: item placement, item bidding and processing</em></figcaption></figure><p>The above event stream shows FaaS functions on the edge processing part of our application. Real-time stateful operations are handled by stream processors; they interact with the log and with correctness guarantees due to the underlying streaming platform protocol. This model ensures temporal guarantees of stream-table joins, i.e., event time correctness.</p><p>Let’s step through the data flow above and focus on the FaaS functionality:</p><ol><li>When item events are sent to the system, they are received via a FaaS function. This function will validate, enrich and either reject the event or write it into the /auction/items stream</li><li>Similarly, bidding events enter the system via a FaaS function whose job is to validate, enrich and then store the event in the /auction/bids stream</li><li>As events pass through the dataflow, an item-complete event is eventually emitted. This triggers a join reaction, and the output is a series of events to notify bidders and calculate analytics.</li></ol><p>In the above example, FaaS functions 1 and 2 obey principle 1: <em>inband but edge</em>. The final stage of post-processing, notifying users, etc., also applies to principle 1.</p><p>If we wanted to include inband processing (principle 2), it might occur as shown below where step 3 (enrich) will process the item-complete event before passing it onto the item-validated topic. This is because latency sensitivity does not affect system behavior. A user can wait 250ms before receiving an email and associated bid analytics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/0*YRWYtQt7nxEes3b9.png" /></figure><p>Provided the event streaming application obeys the FaaS streaming principles, then these functions can become part of any streaming application. As is generally the rule, an understanding of the use cases, awareness of side effects and a touch of common sense mean that it should be possible to build the systems of tomorrow that provide a rich collaboration between serverless runtimes and the event streaming platform.</p><h3>Next steps</h3><p>The synergy of FaaS and the event streaming platform is a natural fit when we think about domain modeling and organizational needs as they change over time. Whether it’s business processes, data models, technology, cloud or just the organization itself, evolution is inevitable and needs to be embraced. We aren’t quite there yet, but it’s undeniable that FaaS is moving at a fast pace, along with streaming and cloud.</p><p>In the context of building event-driven microservices with Kafka or event streaming apps; they are all names for the same thing. The core concept that enables the functionality is the “event” and the ability to record, replay and react to events. Building systems around this principle is fine for streams processors and clients that speak the event streaming platform protocol; however, FaaS doesn’t. As such, we need to ensure that we FaaS functions are operated in such a way so as not to compromise the stream of events but instead be used for the right type of data workloads. Hence, I developed the Faas principles above.</p><h3>What is next for FaaS?</h3><p>The CNCF Serverless Working Group (which I participate in) is shaping how FaaS will look by formalizing the notion of the CloudEvent (spec v1.0 Oct 2019). There is a grand plan that CloudEvents should be publishable, emit through multiple transports and clouds to be consumed by a destination function that is written in any language. Now let’s just pause for a moment and understand the gravity of this. The barrier to cloud becomes pushed down, language considerations are no longer required and propagation of a CloudEvent can potentially go anywhere within the realms of a sanctioned network security domain.</p><p>We have baked into our ideology that events are the future. Making this a reality will further push us to support them at a greater scale and complexity that works seamlessly with FaaS. The (current) clunky barriers that make FaaS feel like duct tape on driving functionality will be broken down to be part of the streaming story. The net effect will be observability, security and a raft of other beneficial considerations that we need to embrace to ensure we don’t fall into previous trappings.</p><p>After all, it shouldn’t be possible for something to get lost in an ocean of FaaS events or stream processors without understanding lineage and instrumentation-do they need to be event sourced? Many lessons have been learnt in the past; I believe we now know enough to stand on the shoulders of giants.</p><p>Stay tuned for part 4, where I’ll discuss the art of the event streaming application and cover streams, stream processors and scale.</p><p>To learn more about event-driven systems, design and concepts I highly recommend Martin Kleppmann’s <a href="https://dataintensive.net/">Designing Data-Intensive Applications</a>, and <a href="https://www.buzzsprout.com/episodes/954362-the-future-of-serverless-and-streaming-with-neil-avery">The Future of Serverless and Streaming</a> podcast.</p><p>If you’d like to know more, you can <a href="https://www.confluent.io/download/">download the Confluent Platform</a>, the leading distribution of Apache Kafka.</p><p>Other articles in this series:</p><ul><li><a href="https://medium.com/@neil.avery_68603/journey-to-event-driven-part-2-programming-models-for-the-event-driven-architecture-eded219630a2">Journey to Event Driven — Part 2: Programming Models for the Event-Driven Architecture</a></li><li><a href="https://medium.com/@neil.avery_68603/journey-to-event-driven-part-3-the-affinity-between-events-streams-and-serverless-apache-84bd37d0f9b9">Journey to Event-Driven — Part 3: The Affinity Between Events, Streams and Serverless</a></li><li><a href="https://medium.com/@neil.avery_68603/journey-to-event-driven-part-4-four-pillars-of-event-streaming-microservices-apache-kafka-9e88fd520a07">Journey to Event Driven — Part 4: Four Pillars of Event Streaming Microservices</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=84bd37d0f9b9" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>