<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Srinivasa on Medium]]></title>
        <description><![CDATA[Stories by Srinivasa on Medium]]></description>
        <link>https://medium.com/@srigumm?source=rss-9c73a885901c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*7pWtRz4EEAhIE075Lf1EWA.png</url>
            <title>Stories by Srinivasa on Medium</title>
            <link>https://medium.com/@srigumm?source=rss-9c73a885901c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 13:39:52 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@srigumm/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[“eCupcakesFactoryOnContainers” — A Reference app for event driven containerized micro services…]]></title>
            <link>https://medium.com/@srigumm/ecupcakesfactoryoncontainers-a-reference-app-for-event-driven-containerized-micro-services-eb845e09ce59?source=rss-9c73a885901c------2</link>
            <guid isPermaLink="false">https://medium.com/p/eb845e09ce59</guid>
            <category><![CDATA[dotnet-core]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[mongo]]></category>
            <category><![CDATA[kafka-connect]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Srinivasa]]></dc:creator>
            <pubDate>Thu, 23 May 2019 14:43:27 GMT</pubDate>
            <atom:updated>2019-05-23T14:43:27.996Z</atom:updated>
            <content:encoded><![CDATA[<h3>“eCupcakesFactoryOnContainers” — A Reference app for event driven containerized micro services architecture using .Netcore and Kafka</h3><h3>INTRODUCTION:</h3><p>This is a continuation of my previous blog posts on <a href="https://medium.com/@sgummadidala/building-realtime-streaming-applications-using-net-core-and-kafka-ad45ed081b31"><strong><em>“Building Real-time Streaming Apps using .Net core and Kafka”</em></strong></a><strong><em>.</em></strong></p><p>Libraries like librdkafka and confluent-kafka-dotnet have opened up Kafka to .NET world, however, there don’t seem to be many examples available. As part of our digital transformation training suite, we started implementing this sample .Net Core reference application — “<a href="https://github.com/srigumm/eCupcakesFactoryOnContainers"><strong>eCupcakesFactoryOnContainers</strong></a><strong>”</strong> based on a simplified “<strong>Microservices</strong>” architecture using docker <strong>containers</strong>. Core intent of this reference application is to provide reference implementation for .Net developers to easily get started with containerized microservices using <strong>.Net Core</strong> and <strong>Kafka</strong>.</p><h3>The Domain:</h3><h4><strong>What is eCupcakesFactory ??</strong></h4><p>It’s a fictional cupcake factory use case mentioned in my colleague <a href="https://medium.com/u/ea7db4409d66">William Scott</a>’s <a href="https://blog.usejournal.com/why-are-you-still-doing-batch-processing-etl-is-dead-3dac2392a772">blog post</a> on event streaming, please read his blogpost before you start looking into this sample codebase.</p><p>To simplify things, our focus will be only on just ordering “<strong>Cupcakes</strong>” with very few flavors, modifying the order and cancelling the order scenarios are out of scope for this blog. When users visit our eCupcakeFactory application, they could order one or more cupcakes, and the kitchen staff would immediately get notified about these new orders. Below are the main steps involved in preparing user order:</p><ul><li><strong>Mixing</strong>: Involves preparing the cupcake batter with right ingredients.</li><li><strong>Baking</strong>: once the mixed batter is available, we pour it in evenly into liners and bake cupcakes in a preheated oven.</li><li><strong>Decorating</strong>: once cupcakes are completely cooled, we can frost and decorate them as simply or creatively.</li><li><strong>Packaging/Boxing</strong>: packaging the right size box and packaging the order.</li><li><strong>Shipping</strong>: delivering the order.</li></ul><h3><strong>Architecture:</strong></h3><p>It’s an event driven architecture implementation with multiple <strong>micro-services</strong>(some are background services) each having its own responsibility and configured to respond on certain events like “NewOrderRecieved”,”OrderBaked”,”OrderMixed” etc.</p><ul><li>Microservices implemented with best coding patterns/practices like CQRS(using MediatR Framework),multi route WebApi controllers etc.</li><li>Used Kafka as message broker for communication b/w these micro services, also leveraged Kafka as <strong>persistent datastore.</strong></li><li>Used confluent’s <strong>“kafka connect”</strong> sink and CDC plugins to import/export data to/from <strong>“MongoDB”.</strong></li><li><strong>React-Redux</strong> architecture for client apps, used http as the communication protocol between the client react apps and the microservices.</li><li>Used <strong>SignalR hubs </strong>for notifying react client applications about the data updates.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LbU3tLZEQ6LFjKo_HqZK2A.png" /></figure><h3><strong>Technology Stack:</strong></h3><ul><li>.Net Core</li><li>Confluent Kafka Cloud</li><li>Confluent Kafka Connect</li><li>SingalR</li><li>React-Redux</li><li>Docker</li></ul><h3>Prerequisites:</h3><ul><li>VSCODE or some .net code editor</li><li>.NET Core 2.2</li><li>Docker&amp;Docker-Compose</li><li>Kafka Installation and Topics setup</li><li>Kafkacat command line tool</li></ul><h3>Git Repository:</h3><p><a href="https://github.com/srigumm/eCupcakesFactoryOnContainers">srigumm/eCupcakesFactoryOnContainers</a></p><h3>Before we go into detail…</h3><h4>Why Kafka for event streaming?</h4><p>We could use other pub-sub/messaging systems like RabbitMQ , but we preferred <strong>Kafka</strong> because of its powerful core features like durability, high throughput, high availability , mutli-consumer model etc.</p><h4>Why .NET Core?</h4><p>we have used .NET Core as it comes with original powerful .net framework and the cross-platform ability to run anywhere like Linux or Windows containers.</p><p>Kafka Confluent’s team is actively working on <a href="https://github.com/confluentinc/confluent-kafka-dotnet">Kafka .NET Core client</a>, releasing every capability in librdkafka(core c library) into this client. Current version v1.0.0 looks very stable with well written patterns.</p><h4>Why SingnalR?</h4><p><strong>SignalR</strong> will give you a much higher level API than raw sockets which is its big advantage, allowing you to do things like “RPC” from server to clients in a broadcast (or targeted) fashion.</p><p>**we couldn’t find any browser based KAFKA client that can stream Kafka messages directly into our react application using <strong>“Consumer Group Semantics”</strong>. so we have implemented a .Net core SignalR based Proxy similar to <a href="https://github.com/microsoft/kafka-proxy-ws">microsoft’s nodejs version</a>.**</p><h3>Setting up and Deployment:</h3><h4><strong>In Local Machine using Docker:</strong></h4><p>To make things simple, we have created a docker compose file with all required apps&amp;services, so running “<strong><em>docker-compose up</em></strong>”command at root directory is enough to start the whole system ( it builds all required docker images and containers in localbox).</p><p>on successful run, you can access your UI/Services at the below endpoints:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e9fd77dd574d1ce51f606fc25c243807/href">https://medium.com/media/e9fd77dd574d1ce51f606fc25c243807/href</a></iframe><p>Note: Above setup assumes that you have a running Kafka version in local machine on standard ports.</p><h3><strong>User Scenarios &amp; Flow of events:</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SHXOXb3-JpSNYaw63GF1Xg.jpeg" /></figure><p>How do we make a cup cake?</p><ul><li>Customer will place an order using our web client (React).</li><li>Mixer(a cook who is also using our web client interface) in the kitchen gets notified about the new order, he/she prepares the mix and updates the order status as “<strong>mixed</strong>”.</li><li>As the order is mixed, Its baker that gets notified about this , he/she pours batter, bakes in oven and updates the order status as “<strong>baked</strong>”.</li><li>Now, decorator gets notified about the baked order , he/she decorates them creatively and updates the order status as “<strong>decorated</strong>”.</li><li>Now, packaging team gets notified about the baked order , they pack the order in the right size box and updates the order status as “<strong>packaged</strong>”.</li></ul><h3>Apps/Services:</h3><h4>eCupcakesFactory Web Client — (React-Redux ui):</h4><p>A react-redux based web application with customers and cooks as its users.</p><ul><li>provides interface to place new orders.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bqoUWqsINi1a1EMGlpMkyg.png" /></figure><ul><li>provides interface for kitchen staff(mixer/baker/decorator/packaging-team) to view newly arrived orders and update the current order status after they complete their process.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8__pA_CSTn05S5CROTMm5A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nbA6xKRwtAsessBbUV4DGQ.png" /></figure><h4>API Layer (REST):</h4><p>Provides the below APIs to place new order and update the current orders.</p><ul><li>api/v1/order → (to place a new order)</li><li>api/v1/order/mix → (to update the current order as mixed)</li><li>api/v1/order/bake → (to update the current order as baked)</li><li>api/v1/order/decorate →(to update the current order as decorated)</li><li>api/v1/order/box →(to update the current order as packaged)</li></ul><h4>Background Services(with SignalR server):</h4><p>.NetCore hosted services continuously monitors KAFKA topics and push notifications to all connected clients(i.e react-ui here) about the newly received orders/updates to existing orders.</p><ul><li>Each of these hosted services has SignalR server with a defined endpoint and actions, UI subscribes to these actions to get notified about the updates to the order.</li></ul><p><strong><em>Example</em></strong>: When a new order arrives in our system, an event gets posted to “<strong>orderrequests</strong>” topic, and our mixer background service which is continuously monitoring this topic gets this message and it uses SignalR server to inform subscribed/connected clients about this.</p><p>Note: why do we need SignalR here? Though Kafka is pub/sub system, there is no direct browser client available. So we need something like SignalR to push these notifications to UI clients.</p><p>our react ui subscribes to the following events of our SignalR endpoints:</p><ul><li>“InformNewOrderToMix”</li><li>“InformNewOrderToBake”</li><li>“InformNewOrderToDecorate”</li><li>“InformNewOrderToPackage”</li></ul><h4>KAFKA Topics:</h4><ul><li>“<strong>orderrequests</strong>” → all received orders</li><li>“<strong>readytobake</strong>” → orders that were mixed and ready to be baked.</li><li>“<strong>readytodecorate</strong>” → orders that were baked and ready to be decorated.</li><li>“<strong>readytopackage</strong>” → orders ready to be packed</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KYdsFFZ3TdqOZaQayktPEQ.jpeg" /></figure><h3>Conclusion:</h3><p>The “<strong>eCupcakesFactoryOnContainers</strong>” is our open-source reference implementation with some patterns (Connect,Streaming topics,CQRS) that we’ve used while implementing .NET Core micro-services using Kafka . You can refer our <a href="https://github.com/srigumm/eCupcakesFactoryOnContainers"><strong><em>github</em></strong></a> repository README files for detailed explanation of each Api/Service/Ui components.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eb845e09ce59" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[KAFKA Security — Configuring SASL Authentication on .NET Core Apps]]></title>
            <link>https://medium.com/tribalscale/kafka-security-configuring-sasl-authentication-on-net-core-apps-da5d0b0fcc5?source=rss-9c73a885901c------2</link>
            <guid isPermaLink="false">https://medium.com/p/da5d0b0fcc5</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[kerberos]]></category>
            <category><![CDATA[thought-leadership]]></category>
            <category><![CDATA[dotnet-core]]></category>
            <category><![CDATA[development]]></category>
            <dc:creator><![CDATA[Srinivasa]]></dc:creator>
            <pubDate>Thu, 04 Apr 2019 15:51:26 GMT</pubDate>
            <atom:updated>2019-04-04T15:51:26.149Z</atom:updated>
            <content:encoded><![CDATA[<h3>Kafka Security: Configuring .Net Core Applications for SASL/GSSAPI (Kerberos) Authentication</h3><p>By: Srinivasa Gummadidala</p><h3>INTRODUCTION:</h3><p>This is a continuation of my previous post on <a href="https://medium.com/@sgummadidala/building-realtime-streaming-applications-using-net-core-and-kafka-ad45ed081b31"><strong><em>“Building Real-time Streaming Apps Using .NET Core and Kafka”</em></strong></a><strong><em>.</em></strong> In this post, we are going to look at the security aspects of Kafka at a high level. We will also configure a .NET Core application to authenticate with a Kerberos-aware Kafka Cluster.</p><h4>Kafka Security Overview:</h4><p>Enabling security over Kafka is optional. However, if your organization is using Kafka to store critical data, enabling Kafka’s security features is crucial to protect your data from all kinds of cyber-attacks.</p><p>Four key security features that you can enable over Kafka:</p><ol><li>Configure the Kafka broker to authenticate internal/external clients using either SASL or Transport Layer Security (TLS) client certificates so that Kafka brokers know who is making each request. Currently, SASL mechanisms are <strong>GSSAPI (Kerberos)</strong> and <strong>PLAIN</strong> are supported.</li><li>Configure “Access Control Lists(ACLs)” to authorize clients read/write access to topics.</li><li>Configure encrypted network communications between producer/consumer and Kafka.</li><li>Configure secure communication between Kafka brokers and ZooKeeper</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/0*5ukWyMfwL3vKpMLx.png" /><figcaption>Kafka Security Overview</figcaption></figure><p>Let’s assume that your organization’s KAFKA instance is configured to authenticate internal/external clients using the <strong>SASL GSSAPI (Kerberos) </strong>option, and assume your organization already has a Kerberos server internally (maybe using Active Directory).</p><p>So, any application/service that needs to communicate with Kafka should get authenticated using SASL(Kerberos).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2carqyvN2OSA6vECNifk5w.png" /></figure><h4>What is SASL?</h4><p>Simple Authentication Security Layer (<a href="https://www.isode.com/products/sasl.html">SASL</a>) is a framework that can be used with other protocols such as Kerberos, SMTP, etc. The basic idea is that an authentication mechanism is separated from the protocol.</p><p>GSSAPI is a SASL authentication mechanism for supporting Kerberos authentication.</p><h4>What is Kerberos?</h4><p>Kerberos is a network authentication protocol based on secret key cryptographies. In simple terms, instead of sharing passwords, communication partners share a cryptographic key, and they use knowledge of this key to verify one another’s identity. You can read more about Kerberos <a href="https://web.mit.edu/kerberos/">here</a>.</p><p>SASL/GSSAPI is a great choice for enterprises as it allows the companies to manage security from within their Kerberos Server.</p><h3>User Scenario:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*sO5fsdEi2X0Ao2KlR_66oA.png" /></figure><p>Let’s take our e-commerce use-case mentioned in our <a href="https://medium.com/@sgummadidala/building-realtime-streaming-applications-using-net-core-and-kafka-ad45ed081b31">previous Kafka blog</a>.<em> </em>Both<em> </em><strong>“Order API”</strong> and <strong>“ProcessOrdersService”</strong> are .NET Core apps that produce/consume messages to/from Kafka topics.</p><p>With your organization using secured Kafka, these two apps need to use Kerberos tokens to get authenticated to the Kafka cluster.</p><h4>Configure .NET Core Producer/Consumer for SASL/Kerberos Authentication:</h4><p>Confluent’s .NET core libraries make configuration easy and there aren’t any extra steps. All that you need to do is generate a <strong>KeyTab file </strong>with your Kerberos principal (user alias in the network) and provide that as part of producer/consumer configuration settings.</p><p><strong>Step 1:</strong> <em>Create or update </em><a href="https://web.mit.edu/kerberos/krb5-1.12/doc/admin/conf_files/krb5_conf.html"><strong><em>krb5.conf</em></strong></a><em> file in /etc/ path.</em></p><ul><li><strong>krb5</strong>.<strong>conf file</strong> contains <strong>Kerberos configuration</strong> information, including the locations of KDCs. Make sure that your organization’s KDC address is configured here.</li><li>Normally, you should install your krb5.conf file in the directory /etc. You can override the default location by setting the environment variable <strong>KRB5_CONFIG</strong></li></ul><p><em>Example:</em></p><blockquote><em>MYORGANIZATION.COM{</em></blockquote><blockquote><em>kdc = testkdcserver.myorg.com</em></blockquote><blockquote><em>}</em></blockquote><p><strong>Step 2:</strong> <em>Generate a KeyTab file</em></p><p>It’s basically a file that contains a table of user accounts with an encrypted hash of the user’s password.</p><p><strong>How do I generate a KeyTab file?</strong></p><ul><li>Use <a href="https://www.ibm.com/support/knowledgecenter/en/SSGSMK_7.1.1/management_sym/sym_kerberos_creating_principal_keytab.html"><strong><em>ktutil</em></strong></a> command</li><li>Create a KeyTab file for each encryption type you use with the <strong>add_entry</strong> command.</li></ul><blockquote><em>add_entry -password -p </em><strong><em>principal_name</em></strong><em> -k number -e </em><strong><em>encryption_type</em></strong></blockquote><ul><li>Once you have created the entries for the principal, write to a KeyTab file.</li></ul><blockquote><em>wkt filename.keytab</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NLvNjG-s1vUOZLJ0txRCrQ.png" /><figcaption>for mac</figcaption></figure><p><strong>How do I verify whether my KeyTab file really works or not?</strong></p><p>Run the below command:</p><blockquote>kinit <a href="/service/mailto:username@MYDOMAIN.COM">username@MYDOMAIN.COM</a> -k -t username.keytab</blockquote><p>You should successfully authenticate without being prompted for a password. Success!</p><p><strong>Step 3: </strong>Update producer/consumer config settings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PETY0ij4UdPXJuo1ETdzFg.png" /></figure><p><strong>Step 4:</strong> Run your application and try producing/consuming messages to Kafka.</p><h4><strong>Troubleshooting Tips:</strong></h4><ul><li>If you are deploying your app in Cloud Containers like Pivotal Cloud Foundry (PCF), make sure that your Docker image comes with SASL libraries. For example, we had an issue with PCF where the cf Linux image that we used to spin a cloud container did not have all the required SASL libraries for Kerberos. One of our TribalScale developers raised a <a href="https://github.com/cloudfoundry/cflinuxfs3/pull/5">merge request</a> in Cloud Foundry to add libsasl2-modules-gssapi-mit package — the request has been approved and merged</li><li>Make sure that your krb5.conf file is in the default path(/etc) or explicitly specify this file location by KRB5_CONFIG environment variable.</li><li>Make sure that you have at least read permissions over the krb5.conf file.</li><li>Producer/consumer classes never surface runtime exceptions related to connectivity directly, they let you instantiate the objects even if you pass invalid credentials. Sometimes, producer/consumer classes might not function as expected without surfacing the underlying exception. You will need to register/attach your error handler to “<strong>OnError</strong>” event to surface these errors.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yf67pTEnDANoIr_T5rGDqA.png" /></figure><ul><li>Keep your KeyTab file at the project root or specify the absolute path in the config settings.</li></ul><p>Note: If you are a windows user, setup <a href="https://docs.microsoft.com/en-us/windows/wsl/install-win10"><strong>WSL</strong></a> (Windows Subsystem for Linux) and try the above steps from ubuntu shell.</p><h3><strong>CONCLUSION</strong>:</h3><p>Enabling Kafka security features is very important and will protect your organization from cyber-attacks. To authenticate Kafka clients, configuring Kafka brokers with SASL/GSSAPI(Kerberos) is also a great choice for enterprises as it allows for security management within the Kerberos Server. Finally, Confluent’s .NET Core APIs have well-defined interfaces that neatly configure .NET Core apps so they can be authenticated to a SASL/GSSAPI(Kerberos)-enabled broker.</p><p>The Kafka community added a number of features that can be used, together or separately, to secure a Kafka cluster. Refer to the <a href="http://Kafka community added a number of features that can be used, together or separately, to secure a Kafka cluster.">Confluent</a> documentation for more details.</p><p>For detailed code, please refer my <a href="https://github.com/srigumm/dotnetcore-kafka-integration">GitHub repository.</a></p><h4>References:</h4><ul><li><a href="https://docs.confluent.io/3.0.0/kafka/sasl.html#kafka-sasl-kerberos-broker">https://docs.confluent.io/3.0.0/kafka/sasl.html#kafka-sasl-kerberos-broker</a></li><li><a href="https://social.technet.microsoft.com/wiki/contents/articles/36470.kerberos-keytabs-explained.aspx">https://social.technet.microsoft.com/wiki/contents/articles/36470.kerberos-keytabs-explained.aspx</a></li><li><a href="http://www.itadmintools.com/2011/07/creating-kerberos-keytab-files.html">http://www.itadmintools.com/2011/07/creating-kerberos-keytab-files.html</a></li><li><a href="http://web.mit.edu/kerberos/krb5-current/doc/user/user_config/kerberos.html#kerberos-7">http://web.mit.edu/kerberos/krb5-current/doc/user/user_config/kerberos.html#kerberos-7</a></li></ul><p><a href="https://www.linkedin.com/in/srigumm/">Srini</a> is an Agile Transformation Engineer at TribalScale based out of Boston office, .NET web developer focused on micro-service-first architecture design.</p><p><a href="https://www.tribalscale.com/"><em>TribalScale</em></a><em> is a global innovation firm that helps enterprises adapt and thrive in the digital era. We transform teams and processes, build best-in-class digital products, and create disruptive startups. Learn more about us on our website. Connect with us on </em><a href="https://twitter.com/tribalscale"><em>Twitter</em></a><em>, </em><a href="https://www.linkedin.com/company/tribalscale/"><em>LinkedIn</em></a><em> &amp; </em><a href="https://www.facebook.com/tribalscale"><em>Facebook</em></a><em>!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da5d0b0fcc5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tribalscale/kafka-security-configuring-sasl-authentication-on-net-core-apps-da5d0b0fcc5">KAFKA Security — Configuring SASL Authentication on .NET Core Apps</a> was originally published in <a href="https://medium.com/tribalscale">TribalScale</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building realtime streaming applications using .NET Core and KAFKA]]></title>
            <link>https://medium.com/@srigumm/building-realtime-streaming-applications-using-net-core-and-kafka-ad45ed081b31?source=rss-9c73a885901c------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad45ed081b31</guid>
            <category><![CDATA[thought-leadership]]></category>
            <category><![CDATA[hosted-services]]></category>
            <category><![CDATA[dotnet-core]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Srinivasa]]></dc:creator>
            <pubDate>Mon, 18 Mar 2019 15:48:12 GMT</pubDate>
            <atom:updated>2019-03-18T17:18:14.815Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building Real-time Streaming Applications Using .NET Core and Kafka</h3><p>By: Srinivasa Gummadidala</p><h3>INTRODUCTION:</h3><p>In the world of MicroServices, .NET core gained more popularity because of its powerful original .NET Framework and ability to run in any platform.<br>The purpose of writing this post is to illustrate on how to use the <a href="https://github.com/confluentinc/confluent-kafka-dotnet">Confluent .NET Core</a> libraries to build streaming applications with Kafka.</p><h4>What is Kafka??</h4><p>As you might have already heard, Kafka is currently the most popular platform for distributed messaging or streaming data.</p><p>The key capabilities of Kafka are:<br> -Publish and subscribe to streams of records<br> -Store streams of records in a fault tolerant way<br> -Process streams of records in real-time<br> -Replayable and real-time</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*brLQW_K4WrLhFKCTdmsr-w.png" /></figure><blockquote>In the Kafka world, a <strong>producer</strong> sends messages(records) to a Kafka node (<strong>broker</strong>) and these messages get stored in a <strong>topics</strong> and <strong>consumers</strong> subscribe to the topics to recieve new messages.</blockquote><p>Kafka is popular among developers because it is easy to pick up and provides a powerful event streaming platform complete with just 4 APIs:<br> — Producer<br> — Consumer<br> — Streams<br> — Connect</p><h3>User Scenario:</h3><p>Let’s take a simple use case of an e-commerce company. Assume we are building a simple <strong>“Order management”</strong> APIs to sell products like “Unicorn Whistles.”</p><p>Our objective here is to <em>build fast &amp; scalable backend APIs to take more order requests and quickly process or trigger other workflows to speed up the delivery process.</em></p><p>To address scaling individual apps and other performance related key metrics, let’s assume that we have decided to build the below two critical components:</p><p>— An Order API(<strong>RESTful</strong>) that takes users orders and responds immediately with some acknowledgement info.<br> — A background service that actually process these order requests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XiyTscILnX5tRoAtcKzSPQ.png" /></figure><h3>Prerequisites:</h3><ul><li>VSCODE or some .net code editor</li><li>.NET Core 2.1</li><li>Docker</li><li>Kafka Installation and Topics setup</li><li>Kafkacat command line tool</li></ul><h3>Installing Kafka and its dependencies:</h3><ul><li>To install Kafka locally using Docker, please follow the instructions <a href="https://github.com/TribalScale/kafka-waffle-stack">here</a>.</li></ul><h3>Creating .NET Core producer:</h3><p>In our use scenario, Order API takes user’s order requests and stores them in a topic called “<strong>OrderRequests.</strong>”<br>So this Order API is going to be one of the producers for the topic “<strong>OrderRequests.</strong>”</p><p>Let’s build OrderAPI as a .Net Core WebApi and add some code to produce messages to the Kafka topic.</p><p><strong>Step 1</strong>: Create a base template for web API<br> — Create a directory with the name “OrderHandler” and run the below commands.</p><blockquote><em>dotnet new webapi -n Api<br>dotnet new xunit -n Test<br>dotnet add Test/ reference Api/</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IMMG2nszjb9S9mUWQt10dQ.png" /><figcaption>Project Structure</figcaption></figure><p><strong>Step 2:</strong> Create OrderController and implement POST handler to receive user’s order requests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GPhIDUoWikdizeAH3mTEVg.png" /></figure><p>Note: Here, I have created a new POCO class called OrderRequest to represent the user’s order request. Refer my <a href="https://github.com/srigumm/dotnetcore-kafka-integration"><strong>github</strong></a> repository for the complete code.</p><p><strong>Step 3</strong>: Install package for Kafka “Confluent.Kafka”</p><blockquote><em>dotnet install package “Confluent.Kafka”</em></blockquote><p>With this, you get access to the producer and consumer APIs and its dependency classes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U5sZA4aQaVnVJYvZoPpL9g.png" /></figure><p><strong>Step 4</strong>: Use Confluent’s “<strong>Producer</strong>” class to connect and produce messages to KAFKA.</p><ul><li><strong>How do we open the connection to Kafka brokers? </strong>Creating a producer class instance with the recommended settings will maintain a connection with all the brokers in the cluster. You should generally avoid creating multiple Consumer or Producer instances in your application.</li></ul><blockquote><em>new Producer&lt;string, string&gt;(this._config)</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XrZpymaodvAqPL3kaTrp1Q.png" /></figure><ul><li><strong>How do we produce messages to topics: </strong>Use<strong> ProduceAsync</strong>(…) async method to write messages to one or more topics and await on the result.</li></ul><p><em>Note: Make sure that you import the </em><em>Confluent.Kafka and </em><em>Confluent.Kafka.Serialization namespaces. Because you need these namespaces for accessing Kafka APIs.</em></p><p>To do things in OOP way and maintain clean code, wrapper the above code inside a new wrapper class(Here I named it as “<strong>ProduceWrapper</strong>”). Refer to my <a href="https://github.com/srigumm/dotnetcore-kafka-integration">github repo</a></p><h3>RUN and TEST Producer:</h3><p>Make sure that you invoke the producer wrapper code from your controller code as below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VQ3tC37HP751yhU-5guDKg.png" /></figure><p>Issue <em>“</em><strong><em>dotnet run</em></strong><em>”</em> command to start the web server.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NEf8MOQL68QHfeLJl4LbBA.png" /></figure><p>Trigger user order requests from POSTMAN or any rest client:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f1xvJSol9NC6sFoowBbPDg.png" /></figure><p>Verify if our requests were produced to Kafka’s “<strong>OrderRequests</strong>” topic :</p><ul><li>Use “<strong>kafkacat</strong>” command line utility tool to verify</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4mg6piEpV4GHW5IktY0zYQ.png" /></figure><h3>Creating .NET Core Consumer:</h3><p>Use Confluent’s “Consumer” class to connect and consume message from Kafka topic.</p><p><strong>How do we open the Kafka connection and consume messages?</strong></p><ul><li>Create a consumer class instance with the recommended settings and subscribe to the topic.</li></ul><blockquote>var consumer = new <strong>Consumer</strong>&lt;string,string&gt;(this._consumerConfig);</blockquote><blockquote>consumer.<strong>Subscribe</strong>(topicName);</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZabSNVpOgukHjTAmQrb5uA.png" /></figure><ul><li>Use “Consume()” method to start reading messages from the topic.</li></ul><p><strong>ConsumerConfig Settings:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*a403Mf5XitgYTwMxQSXIaA.png" /></figure><ul><li><strong>GroupId</strong>: Records will be load balanced between consumer instances with the same group id.</li><li><strong>AutoOffsetReset</strong>: Kafka lets you consume topic records in any order, you can consume from the beginning or latest, or reset to a particular position, and so on.</li><li><strong>EnableAutoCommit</strong>: Kafka uses a concept called OFFSET to track each consumer position in the entire topic consumption. When this setting is on, your consumer instance is going to commit its Offset to Kafka every time you fetch a record.</li></ul><p>As we have already implemented a POST handler to capture user order requests into “orderrequests” Kafka topic, now we need to build something to process these records and write to “<strong>readytoship</strong>” Kafka topic. And this should happen in real time means as soon as records arrive in orderrequest topic.</p><p>We can create a background service that streams records from “orderrequests” topic.</p><p><strong>Step 1</strong>: Create a new <a href="https://dotnetcoretutorials.com/2019/01/13/hosted-services-in-asp-net-core/">.NET Hosted Service </a>called <strong>“ProcessOrdersService”</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ucaVZZ_cSPxepudUf6p8qg.png" /></figure><p>Step 2: Use Confluent library’s <strong>Consumer</strong> class to read messages from topic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4hws7bjVwcZresrBQJKmtQ.png" /></figure><h3>RUN and TEST OrderProcessing service:</h3><p>.NET Core lets you to host background services inside webhost or generic host, please refer this <a href="https://www.stevejgordon.co.uk/using-generic-host-in-dotnet-core-console-based-microservices">article</a>.</p><p>Since we already have a webhost(created for webAPI), keeping our HostedService inside webAPI project will be sufficient to run this service.</p><p>“dotnet run” command will start both web server and this background service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/825/1*IgjlA8PU5u-w2FTwZrkTcA.png" /><figcaption>project structure</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J2pPmXAfBUF4MpAI-o93WA.png" /><figcaption>.NET Core HostedService</figcaption></figure><p>Verify if messages are produced to “readytoship” topic using Kafkacat utility.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sa13Ifmj6-M9dK9o4lYFkQ.png" /></figure><h3><strong>CONCLUSION</strong>:</h3><p>Using Confluent’s .NET Producer and Consumer APIs is simple and straightforward, which makes it easy to adopt them for real microservices/streaming apps.</p><p>For detailed code , please refer my <a href="https://github.com/srigumm/dotnetcore-kafka-integration">githhub repository.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/340/1*3-jIShiwV5tto1QsBmQ-rQ.png" /></figure><p><a href="https://www.linkedin.com/in/srigumm/">Srinivas</a> is an Agile Transformation Engineer at TribalScale based out of Boston office, .NET web developer focused on micro-service-first architecture design.</p><p><a href="https://www.tribalscale.com/"><em>TribalScale</em></a><em> is a global innovation firm that helps enterprises adapt and thrive in the digital era. We transform teams and processes, build best-in-class digital products, and create disruptive startups. Learn more about us on our website. Connect with us on </em><a href="https://twitter.com/tribalscale"><em>Twitter</em></a><em>, </em><a href="https://www.linkedin.com/company/tribalscale/"><em>LinkedIn</em></a><em> &amp; </em><a href="https://www.facebook.com/tribalscale"><em>Facebook</em></a><em>!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad45ed081b31" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>