<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Rick Osowski on Medium]]></title>
        <description><![CDATA[Stories by Rick Osowski on Medium]]></description>
        <link>https://medium.com/@rosowski?source=rss-5adc9417d038------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*j4TzvVx0QRctkBWy3YIMfg.jpeg</url>
            <title>Stories by Rick Osowski on Medium</title>
            <link>https://medium.com/@rosowski?source=rss-5adc9417d038------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 15:52:34 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rosowski/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Kafka Security Fundamentals — adding TLS to your event-driven utility belt!]]></title>
            <link>https://rosowski.medium.com/kafka-security-fundamentals-adding-tls-to-your-event-driven-utility-belt-432307f4ff62?source=rss-5adc9417d038------2</link>
            <guid isPermaLink="false">https://medium.com/p/432307f4ff62</guid>
            <category><![CDATA[event-streaming]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[event-driven-architecture]]></category>
            <category><![CDATA[event-driven-systems]]></category>
            <dc:creator><![CDATA[Rick Osowski]]></dc:creator>
            <pubDate>Fri, 04 Jun 2021 17:10:22 GMT</pubDate>
            <atom:updated>2021-06-04T17:10:22.772Z</atom:updated>
            <content:encoded><![CDATA[<h3>Kafka Security Fundamentals — adding TLS to your event-driven utility belt!</h3><h3>Getting Started… again!</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*opsIDKwxOkRYn5UT.png" /><figcaption>Apache Kafka project official logo</figcaption></figure><p>Understanding <a href="https://kafka.apache.org/"><strong>Apache Kafka</strong></a> security is much like discovering the Rosetta Stone to the entire event-streaming landscape. In my <a href="https://rosowski.medium.com/kafka-security-fundamentals-the-rosetta-stone-to-your-event-streaming-infrastructure-518f49640db4"><strong>previous blog post</strong></a>, I covered how understanding a few common core configurations provides you the ability to connect any Kafka-based, event-driven application to dozens of Kafka offerings with relatively few changes. However, the previous blog post focused on username and password-based authentication. This blog post will cover those same core configurations but from the perspective of SSL Certificate-based authentication. Or as it is more commonly known, Mutual TLS (<strong>mTLS</strong>)-based authentication.</p><p>As a caveat, this blog post will continue to cover the most important Kafka security configuration options from a Kafka client perspective <em>(based on </em><a href="https://ibm-cloud-architecture.github.io/refarch-eda/"><strong><em>my team’s experiences</em></strong></a><em> so far)</em> to allow you to pull this knowledge from your utility belt like a certain dark knight when it comes to any Kafka offering or security model.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Faj9CDZm0Glc%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Daj9CDZm0Glc&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Faj9CDZm0Glc%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/a569b91e6a70d5d7363bff70bdc95817/href">https://medium.com/media/a569b91e6a70d5d7363bff70bdc95817/href</a></iframe><p>This is a deeper dive into Apache Kafka and its capabilities, so I will not be covering any introductory Kafka material here. You can head over to <a href="https://kafka.apache.org/"><strong>kafka.apache.org</strong></a> for all the foundational information necessary or check out my friend Whitney’s <a href="https://www.youtube.com/watch?v=aj9CDZm0Glc"><strong>What is Kafka?</strong></a> video to learn everything you need to get started in under 10 minutes. Once you’ve read through everything here, if you still have questions or want to dig in further, the <a href="http://kafka.apache.org/documentation/#security"><strong>Security</strong></a> section of the official Kafka documentation covers everything in depth with a great deal of clarity — so that would be my first recommendation if you read through here to find that none of the common configurations below match your expected environment setups!</p><p>Before we jump in, there is one remaining caveat from our last blog post to address:</p><ul><li>There are many, many ways to configure Kafka security from a Kafka administrator point-of-view. The focus of this short blog series is to more effectively comprehend Kafka security as a Kafka client (<em>producer / consumer</em>) and not a Kafka administrator (<em>operations / provider</em>). For a deeper understanding of practical Kafka operational security from an administrator’s point of view, you can reference Confluent’s <a href="https://docs.confluent.io/platform/current/security/general-overview.html"><strong>Security Overview</strong></a> for a much deeper understanding on how on to securely deploy and maintain Kafka well into the future!</li></ul><p>Without any further ado, let’s get started!</p><h3>SSL Certificate Rewind</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FT4Df5_cojAs%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DT4Df5_cojAs&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FT4Df5_cojAs%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/a946d571d55c8fe896438f7d0cb4fda2/href">https://medium.com/media/a946d571d55c8fe896438f7d0cb4fda2/href</a></iframe><p>The <a href="https://www.youtube.com/watch?v=T4Df5_cojAs"><strong>above video</strong></a> is a great introduction <em>(or refresher)</em> for <a href="https://en.wikipedia.org/wiki/Public_key_certificate"><strong>SSL/TLS certificates</strong></a> and the underlying assumptions and properties the majority of our digital security is founded upon. Borrowing the metaphor the speaker in the video uses, the two major assumptions of SSL security <em>(and all </em><a href="https://en.wikipedia.org/wiki/Public-key_cryptography"><strong><em>public-key cryptography</em></strong></a><em>)</em> are the following:</p><ol><li>Any message encrypted with Bob’s public key can only be decrypted with Bob’s private key</li><li>Anyone with access to Alice’s public key can verify that a message could only have been created by someone with access to Alice’s private key.</li></ol><p>For our use case, these two assumptions serve the exact function we need to perform both encryption and authentication-by-identity in our Kafka ecosystem. As such, the Kafka cluster components and your Kafka clients will need to share the necessary SSL/TLS certificates and keys between each other to communicate effectively, in a secured and controlled manner. This can be both complex and utterly confusing at the same time. But hopefully, after watching the video above and reading through the rest of this blog post, it will all be a little clearer.</p><p>With all that said, the minimum set of artifacts required for a Kafka client to authenticate with a Kafka cluster utilizing mTLS authentication is:</p><ul><li><em>SSL Truststore</em>, containing the Certificate Authority of the Kafka cluster <em>(its public key) (or an intermediate certificate used for signing purposes that the Kafka cluster will trust)</em></li><li><em>SSL Truststore password</em></li><li><em>SSL Keystore</em>, containing <strong>(1)</strong> the Certificate Authority of the Kafka cluster <em>(its public key)</em> <em>(or an intermediate certificate used for signing purposes that the Kafka cluster will trust) </em>and <strong>(2)</strong><em> </em>the client certificate key-pair, signed by the Certificate Authority<em> (both the public and private keys of the client)</em></li><li><em>SSL Keystore password</em></li></ul><h3>Kafka Settings</h3><h4><a href="http://kafka.apache.org/documentation/#adminclientconfigs_bootstrap.servers">bootstrap.servers</a></h4><p>This is the most important setting as it tells Kafka clients where to look to talk to your applications. Its context, value, and explanation is unchanged from the previous blog post. Sometimes this will be a string that contains just a single hostname:port combination, while other times it will be a comma-separated list of hostname:port combinations. Depending upon the Kafka offering you are connecting to, as well as how and where it is deployed, you will either be providing a singular bootstrap address or the entire collection of Kafka brokers in the same string. The bootstrap endpoint is a functional intermediary that Kafka offerings provide to allow for simpler networking in cases of expected fluctuations in the number of Kafka brokers, as it is a single URL that will return all the active brokers in the cluster to your clients automatically.</p><h4><a href="http://kafka.apache.org/documentation/#adminclientconfigs_security.protocol">security.protocol</a></h4><p>This is the next most important security setting as it tells the clients how to communicate with the brokers. If you’ve ever had HTTPS issues while using a web browser, this is the setting that would give Kafka clients SSL issues while attempting to connect to Kafka brokers! If you are receiving errors in your application while attempting to connect to Kafka with references to a broker (-1), you will usually need to revisit this setting <em>(i.e. you should be using </em><em>SSL but you configured </em><em>PLAINTEXT, etc.)</em>.</p><p>The previous blog post used the values of SASL_PLAINTEXT or SASL_SSL to tell Kafka we wanted to communicate with SASL-based usernames and passwords. However, if you need to authenticate with mTLS certificates instead, your value for this configuration option will always be <strong>SSL</strong>.</p><p>The valid values are:</p><ul><li>PLAINTEXT <em>(using PLAINTEXT transport layer &amp; no authentication — default value)</em></li><li>SSL <em>(using SSL transport layer &amp; certificate-based authentication)</em></li><li>SASL_PLAINTEXT <em>(using PLAINTEXT transport layer &amp; SASL-based authentication)</em></li><li>SASL_SSL <em>(using SSL transport layer &amp; SASL-based authentication)</em></li></ul><h4><a href="http://kafka.apache.org/documentation/#adminclientconfigs_ssl.truststore.location">ssl.truststore.location</a></h4><p>To allow for encryption of communication between Kafka brokers and clients, as specified by our security.protocol setting above, we need to provide our Kafka clients with the location of a trusted Certificate Authority-based certificate.</p><p>This file is often provided by the Kafka administrator and is generally unique to the specific Kafka cluster deployment. It is outside the scope of this blog post to determine how you provide access to this file for your individual runtimes, but it must exist inside the runtime container somewhere.</p><p>For Java-based clients, this file is usually in the JKS format by default. However, in recent Kafka releases, PKCS12 support has been added for Java clients as well. For Python or NodeJS-based clients, this file should be in either a PEM or PKCS12 format. To extract a PEM-based certificate from a JKS-based truststore, you can use the following command:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/61c0fd6bb4ad2e500f2091bb3f092e4a/href">https://medium.com/media/61c0fd6bb4ad2e500f2091bb3f092e4a/href</a></iframe><h4><a href="http://kafka.apache.org/documentation/#adminclientconfigs_ssl.truststore.password">ssl.truststore.password</a></h4><p>When providing a JKS or PKCS12-based truststore for validation of encryption certificates via the ssl.truststore.location configuration property above, you will need to provide a password to access the truststore file.</p><p>This should be provided by your Kafka administrator at the same time they provide you with the truststore <em>(or at least provide you with a method to acquire the truststore password)</em>.</p><h4><a href="http://kafka.apache.org/documentation/#adminclientconfigs_ssl.keystore.location">ssl.keystore.location</a></h4><p>This setting is where the Kafka client provides its certificate-based authentication credentials to the Kafka brokers, as specified by our security.protocol setting as SSL. We need to provide our Kafka clients with the location of a unique identifying certificate that is signed by the overall Cluster CA.</p><p>This file is often provided by the Kafka administrator or generated as part of initial client or user configuration. It is outside the scope of this blog post to determine how you provide access to this file for your individual runtimes, but it must exist inside the runtime container somewhere.</p><p>For Java-based clients, this file is usually in the JKS format by default. However, in recent Kafka releases, PKCS12 support has been added for Java clients as well. For Python or NodeJS-based clients, this file should be in either a PEM or PKCS12 format.</p><h4><a href="http://kafka.apache.org/documentation/#adminclientconfigs_ssl.keystore.password">ssl.keystore.password</a></h4><p>Similar to ssl.truststore.password, when providing a JKS or PKCS12-based keystore for client authentication purposes via the ssl.keystore.location configuration property above, you will need to provide a password to access the associated keystore file.</p><p>This can be provided by your Kafka administrator at the same time they provide you with the CA truststore <em>(or at least provide you with a method to acquire any additional client keystore/key passwords)</em>.</p><h4><a href="http://kafka.apache.org/documentation/#adminclientconfigs_ssl.key.password">ssl.key.password</a></h4><p>This configuration property contains the value of the password to the private key that is in the keystore file <em>(or the PEM key if specified explicitly in </em><em>ssl.keystore.key, but that is an alternate configuration setting we are not covering in this blog post.)</em> This value is most often the same as ssl.keystore.password above and is explicitly required when clients are to perform two-way authentication, which is our use case here.</p><p>Similar to the other ssl.keystore.* settings, this can be provided by your Kafka administrator at the same time they provide you with the CA truststore <em>(or at least provide you with a method to acquire any additional client keystore/key passwords)</em>.</p><h3>Common configuration options for the most popular offerings</h3><p>As you will see in the sections below, the differences between Kafka offerings when using mTLS-based authentication is minimal. All the technical detail comes in how you are generating the SSL certificates that you are providing to your clients. In other words, the devil is in the details. These sections will still cover the specific settings necessary for each offering type, with additional reference material where appropriate for help in understanding SSL certificate generation for each offering. Comprehensive SSL certificate management for each Kafka offering is outside the scope of this blog post.</p><h4>Strimzi / Red Hat AMQ Streams / IBM Event Streams</h4><p><a href="https://strimzi.io/"><strong>Strimzi</strong></a> is an open-source project which provides a Kubernetes Operator-focused way of deploying Apache Kafka clusters flexibly, repeatedly, and securely to any Kubernetes distribution. The enterprise-grade offerings of <a href="https://www.redhat.com/en/blog/getting-started-red-hat-amq-streams-operator"><strong>Red Hat AMQ Streams</strong></a> and <a href="https://ibm.github.io/event-streams/"><strong>IBM Event Streams</strong></a> are built on top of the underlying Strimzi Operator Custom Resources and take advantage of much of the same configuration specifications.</p><p>The power of the Operator model that the Strimzi variants bring to Kafka is the automated maintenance and operational detail that is otherwise necessary to do explicitly by hand or custom automation. Strimzi provides a UserOperator and a TopicOperator subcomponent for administration of those contextual objects.</p><p>In the example to be covered later, we leverage the UserOperator to automatically generate signed SSL certificates for a new user which are based off of the internal Certificate Authority contained in the Strimzi Kafka deployment. There is no need for manual SSL certificate configuration!</p><p>For information on configuring users and access credentials this way, visit the <a href="https://strimzi.io/docs/operators/latest/using.html"><strong>Strimzi Docs</strong></a> page for <a href="https://strimzi.io/docs/operators/latest/using.html#con-securing-client-authentication-str"><strong>User Authentication</strong></a> as the least-common-denominator between all three noted Strimzi variants.</p><p>External clients connecting to the Kubernetes or OpenShift cluster which Strimzi et al are deployed to often use the following configurations when requiring two-way SSL/mTLS authentication:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3c8165725f8c037711340c9628ed419f/href">https://medium.com/media/3c8165725f8c037711340c9628ed419f/href</a></iframe><p>Clients internal to the Kubernetes or OpenShift cluster which Strimzi et al are deployed to often use the following configurations:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e85314530105247ec34bb3b3f623b313/href">https://medium.com/media/e85314530105247ec34bb3b3f623b313/href</a></iframe><p><strong><em>NOTE:</em></strong> Recent Strimzi versions have added support for OAuth-based authentication, beyond just SASL or mTLS-based authentication. We have not widely encountered this in the field yet, but will address it once we do.</p><p>For more information on the specifics of configuring <a href="https://strimzi.io/docs/operators/latest/using.html#assembly-securing-access-str"><strong>secure listeners on Strimzi-based Kafka clusters</strong></a>, you will want to visit the official <a href="https://strimzi.io/docs/operators/latest/using.html"><strong>Strimzi Docs</strong></a>. The examples in the documentation, as well as our demo scenario below, utilize three listeners with different security profiles <em>(PLAINTEXT vs SSL transport layers / no authentication vs SASL-based authentication vs mutual TLS authentication / internal vs external clients)</em> based upon the expected clients connecting to them.</p><h4>Confluent Platform</h4><p>Confluent provides many configuration options and deployment targets for their Confluent Platform on-premises Kafka offering. Our team’s main focus is Red Hat OpenShift-based deployments and we focus on the capabilities made available to Confluent Platform through the <a href="https://docs.confluent.io/operator/current/overview.html"><strong>Confluent Operator</strong></a>. As such, we focus on <a href="https://docs.confluent.io/operator/current/co-authenticate.html"><strong>SASL and mTLS-based authentication mechanisms with optional TLS-encrypted traffic</strong></a>.</p><blockquote>Confluent has recently released an updated Kubernetes Operator, called <a href="https://docs.confluent.io/operator/current/overview.html"><strong>Confluent for Kubernetes</strong></a>. This blog post will speak to the previous version of the Operator, <a href="https://docs.confluent.io/operator/1.7.0/overview.html"><strong>Confluent Operator 1</strong></a>, to align with the previous blog post in this series. Minimal technical changes are expected between the two Operator versions as they pertain to the use cases discussed here.</blockquote><p>For a deeper dive into the extent of SSL and Mutual TLS security inside of Confluent Platform, you can refer to the robust <a href="https://docs.confluent.io/platform/current/security/security_tutorial.html"><strong>Security Tutorial</strong></a> in the Confluent Docs and specifically the section on <a href="https://docs.confluent.io/platform/current/security/security_tutorial.html#creating-ssl-keys-and-certificates"><strong>Creating SSL Keys and Certificates</strong></a>. For a more streamlined approach to what is minimally required for a valid SSL client certificate in a Confluent Platform environment, you can reference the <a href="https://github.com/confluentinc/confluent-platform-security-tools"><strong>Confluent Platform Security Tools</strong></a> GitHub repository for scripts that automate the creation of the necessary keystores and truststores Kafka clients need.</p><blockquote>For an ultra-streamlined version of the Confluent Platform Security Tools scripts when you already have a cluster CA certificate and key, you can refer to this <a href="https://gist.github.com/osowski/6abf268e9d7ab521481cc35523bc50f6"><strong>gist</strong></a> I adapted from the Confluent Platform Security Tools scripts.</blockquote><p>Clients external to the Kubernetes or OpenShift cluster which Confluent Platform is deployed to most often use the following configurations:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/706df1eb40bc0384e24667f6b4898ece/href">https://medium.com/media/706df1eb40bc0384e24667f6b4898ece/href</a></iframe><p>Clients internal to the Kubernetes or OpenShift cluster which Confluent Platform is deployed to most often use the following configurations:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9ad55ac9586842467c169c91349baf08/href">https://medium.com/media/9ad55ac9586842467c169c91349baf08/href</a></iframe><h4>IBM Event Streams on Cloud</h4><p>IBM provides a hosted, SaaS-based Kafka offering as part of IBM Cloud and it is available via the <a href="https://cloud.ibm.com/catalog/services/event-streams"><strong>IBM Cloud Catalog</strong></a>. It comes pre-configured with everything you need to immediately get started working with Kafka applications, along with authentication that is directly integrated into the IBM Cloud IAM. As such, it does not support SSL-based authentication for clients at this time.</p><h3>Practical Use Case with Kafka binaries</h3><p>Now we will walk through a straight-forward use case of deploying a vanilla open-source Strimzi cluster <em>(which uses the same underlying configuration as IBM Event Streams V10 and Red Hat AMQ Streams)</em>, creating a TLS-based user, creating a topic, and interacting with that topic through the CLI tools provided by the Kafka binaries.</p><p>The only parameters we will be configuring from the downstream client perspective are the ones we covered above. This quick tutorial assumes access to a Red Hat OpenShift 4.5+ cluster. If you do not have one, you can access one easily via the <a href="https://developer.ibm.com/openlabs/openshift"><strong>IBM Open Labs</strong></a>.</p><p><em>(</em><strong><em>NOTE:</em></strong><em> You will want to avoid the OpenShift Playground clusters available on Katacoda for this exercise, as they use a more complex SSL certificate setup than the scope of this article is meant to address.)</em></p><h4>Scenario setup</h4><ul><li>Access an OpenShift 4.5+ cluster via the OpenShift Console.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vU5VJL1ovp6PuWNdovMieQ.png" /></figure><ul><li>Go to Operators on the left-hand side menu and click <strong>OperatorHub</strong>. Then, search for <em>strimzi</em>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BKYo4scFSV6dNrpRoam8VQ.png" /></figure><ul><li>Click the <strong>Strimzi</strong> tile and click <strong>Install</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZnI10Q5BtYhEYcOMzAk9ig.png" /></figure><ul><li>Make sure you are installing from the <strong><em>stable</em></strong> channel and you are installing the operator in <strong><em>All namespaces on the cluster</em></strong>. Click <strong>Install</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PNoNoqZ9ZmxJT5WyhWIi-g.png" /></figure><ul><li>After a couple of minutes, you should see the <strong>Strimzi</strong> operator successfully deployed and running.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i3YEBFejw4S09OL2CAcxGA.png" /></figure><ul><li>Go to <strong>Projects</strong> on the left-hand side menu and click <strong>Create Project</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tn9z1s0xck1J8gewTjm5Ww.png" /></figure><ul><li>Once the project is created, you should be presented with a dashboard for it.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*24OsejpIGD0Eubk9N_3meA.png" /></figure><h4>Create a Kafka cluster instance.</h4><ul><li>Click <strong>Installed Operators</strong> under the Operators section in the left-hand side menu. Click the <strong>Strimzi</strong> operator.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FMPIMEmVOfLXxRARRuWz0w.png" /></figure><ul><li>Click the <strong>Kafka</strong> tab and click <strong>Create Kafka</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fL0DTk7CYl_LKVjHL_oWIQ.png" /></figure><ul><li>Switch to the YAML view.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5chrUTkixd57SAp0GFcnYA.png" /></figure><ul><li>Update the listeners section within the .spec section with the following configuration:</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/25da89bbb88ebac2f02d9088079dd9cc/href">https://medium.com/media/25da89bbb88ebac2f02d9088079dd9cc/href</a></iframe><p>The above <em>listeners</em> configuration will create an internal non-secured listener on port 9092, an internal TLS-secured listener on port 9093, and an external TLS-secured listener on port 9094. We will use this last port to access our Kafka cluster from outside of OpenShift. As a result, we need a TLS certificate for the SSL encryption and authentication.</p><ul><li>Make sure the <strong>entityOperator</strong> along with the <strong>topicOperator</strong> and the <strong>userOperator</strong> are also defined within the <strong>spec</strong> section. If they are not, add them.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rYmlZmP4vM5inaiJbeCKzQ.png" /></figure><ul><li>Once you have updated the listeners section, click <strong>Create</strong>. After a minute or so, you should see the Kafka Cluster ready.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m34mRkH_q1SLFi0PqA5ouQ.png" /></figure><h4>Create a Kafka topic via the Strimzi Operator.</h4><ul><li>Click the <strong>KafkaTopic</strong> tab and then click <strong>Create KafkaTopic.</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XQIUTEoyKiK0aqvNSmXLFw.png" /></figure><ul><li>Leave the defaults and click <strong>Create</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LYNV3gpMuHJlQv755HNA2w.png" /></figure><blockquote><strong>NOTE</strong>: Your Topic Name should stay `my-topic` for this simple example to align with the default ACLs for the KafkaUser you will create next.</blockquote><ul><li>You should see your topic created.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MZKbnKMZNo4p5fgeaAlw0g.png" /></figure><h4>Create a KafkaUser via the Strimzi Operator</h4><ul><li>Click the <strong>KafkaUser</strong> tab and then click <strong>Create KafkaUser</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1h97Wl3QlJijqpI0plNHlg.png" /></figure><ul><li>Click <strong>Authentication</strong> and ensure the authentication type is set to tls. Click <strong>Create</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yohwa_F1SaLscH92oGhAQQ.png" /></figure><p>This will create a Kafka user with a set of TLS certificates so that it can access the Kafka Cluster from outside of OpenShift using the external listener on port 9094 you created earlier.</p><ul><li>You should see the user created.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gb58RT7GpvO-72CguijDOw.png" /></figure><h4>Generate configuration properties</h4><p>On your terminal, execute the following commands.</p><ul><li>Download the Kafka binaries to your sandbox environment.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fe5dc5f5f3cd67eeec0b083e0113cf32/href">https://medium.com/media/fe5dc5f5f3cd67eeec0b083e0113cf32/href</a></iframe><ul><li>Change project to kafka-security .</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/08d3dd8e453c7beb321d4d4ddc706998/href">https://medium.com/media/08d3dd8e453c7beb321d4d4ddc706998/href</a></iframe><ul><li>Export our configuration data to environment variables for easy reuse.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b99a05af4644d35fd030783b837281ad/href">https://medium.com/media/b99a05af4644d35fd030783b837281ad/href</a></iframe><ul><li>Initialize local configuration file with the SSL security protocol.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/23d73ddee60222c34805776915fc6412/href">https://medium.com/media/23d73ddee60222c34805776915fc6412/href</a></iframe><p>Since we are working with the Kafka cluster we deployed earlier from outside of the OpenShift cluster it is deployed on, we need to do that through the external TLS-secured 9094listener. The Strimzi Operator will take care of creating the external access to that listener within the OpenShift cluster in the form of an OpenShift Route. As a result, the security.protocol we need to set in the configuration file that the Kafka binaries scripts will later use is SSL.</p><ul><li>Play the role of the Kafka Administrator, extract your Truststore artifact, and pass in the absolute path to the Java Truststore artifact on the local filesystem.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/21ec9a0abbcc45690c012ce2b582ab04/href">https://medium.com/media/21ec9a0abbcc45690c012ce2b582ab04/href</a></iframe><p>This certificate will allow us to establish SSL-encrypted outbound communication to our Kafka cluster.</p><ul><li>As a result of creating the KafkaUser custom resource, the Strimzi Operator has created a Kubernetes Secret containing all the necessary SSL certificate components you need to authenticate and communicate with the Kafka cluster securely. Extract your Keystore artifact and pass in its absolute path on the local filesystem.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d2961ecb365f56fbba6ae4b37422f6ed/href">https://medium.com/media/d2961ecb365f56fbba6ae4b37422f6ed/href</a></iframe><p>This certificate will allow us to authenticate properly with the Kafka cluster and establish our client’s cryptographically-secured identity.</p><blockquote><strong>NOTE: </strong>The files extracted from the my-user Secret are the necessary SSL certificates and passwords that are automatically generated for us by the Strimzi Operator. There is no need for manual SSL certificate management when using the Strimzi Operator. However, when using the Confluent Platform, you would need to perform the steps documented in the <a href="https://github.com/confluentinc/confluent-platform-security-tools">Confluent Platform Security Tools scripts</a> or my <a href="https://gist.github.com/osowski/6abf268e9d7ab521481cc35523bc50f6">streamlined gist</a> to get those same files.</blockquote><ul><li>View topic data via <strong>bin/kafka-topic.sh</strong> tool:</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5693ebd753f3121c5311476cdda18f0b/href">https://medium.com/media/5693ebd753f3121c5311476cdda18f0b/href</a></iframe><ul><li>Send data with the <strong>bin/kafka-console-producer.sh</strong> tool:</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e171ac6ce6c4385b63646b459ca91105/href">https://medium.com/media/e171ac6ce6c4385b63646b459ca91105/href</a></iframe><blockquote><strong>NOTE:</strong> Use `Ctrl+C` when you are done sending data to exit the console producer.</blockquote><ul><li>Read data with the <strong>bin/kafka-console-consumer.sh</strong> tool:</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/31a99f06a8f90478b39062236a842543/href">https://medium.com/media/31a99f06a8f90478b39062236a842543/href</a></iframe><blockquote><strong>NOTE:</strong> Use `Ctrl+C` when you are done reading data to exit the console consumer.</blockquote><p>With this understanding of Kafka client security configuration, you can now seamlessly switch between Strimzi to IBM Event Streams to Red Hat AMQ Streams to Confluent Platform to IBM Event Streams on Cloud to <a href="https://lenses.io/"><strong>Lenses.io</strong></a> to <a href="https://github.com/obsidiandynamics/kafdrop"><strong>kafdrop</strong></a> to <a href="https://github.com/edenhill/kafkacat"><strong>kafkacat</strong></a><strong> </strong>to <a href="https://github.com/apache/kafka/tree/trunk/bin"><strong>Kafka-provided binaries</strong></a> and beyond by following the same process!</p><h3>Conclusion</h3><p>These same handful of settings will occur over and over again, from Producer Configs to Consumer Configs to Kafka Streams Configs to Kafka Connect Configs to Admin Client Configs to command-line configs. They are everywhere! Depending on your implementation language, platform, and tooling, there may be some assistance available in the SDKs, APIs, or CLIs to prevent the need of copying and pasting across the different Kafka component configurations, but once you get the configuration correct at the highest level, everything else in your event-driven applications seems much simpler to connect, debug, and validate at the lower levels!</p><p>Now with the addition of SSL certificate-based mutual authentication (<strong>mTLS</strong>), you are able to build, configure, and deploy all types of Kafka clients that will easily adapt to security models and infrastructure requirements — whether that is remote credential management or a cloud-provider-based key management system to handle and rotate your certificates regularly.</p><p>I hope this whirlwind tour of practical Kafka security configuration options has been helpful and will become a handy reference for you to use going forward! Please let me know if you find this useful and would like to see more practical Kafka experiences documented here. In the meantime, head over to our <a href="https://www.ibm.com/cloud/architecture/architectures/eventDrivenArchitecture/overview"><strong>Event-Driven Reference Architecture</strong></a> in the <a href="https://www.ibm.com/cloud/architecture/architectures"><strong>IBM Cloud Architecture Center</strong></a> that has the latest and greatest collection of all things event-driven and Kafka-based!</p><p><em>A special shoutout to </em><a href="https://www.linkedin.com/in/jesus-almaraz-hernandez/"><em>Jesus Almaraz</em></a><em> &amp; </em><a href="https://www.linkedin.com/in/jeromeboyer/"><em>Jerome Boyer</em></a><em> for their reviews of, edits on, and polishing contributions to this article.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=432307f4ff62" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kafka Security Fundamentals — The Rosetta Stone to your event-streaming infrastructure]]></title>
            <link>https://rosowski.medium.com/kafka-security-fundamentals-the-rosetta-stone-to-your-event-streaming-infrastructure-518f49640db4?source=rss-5adc9417d038------2</link>
            <guid isPermaLink="false">https://medium.com/p/518f49640db4</guid>
            <category><![CDATA[event-driven-architecture]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[event-streaming]]></category>
            <category><![CDATA[event-driven-systems]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Rick Osowski]]></dc:creator>
            <pubDate>Tue, 27 Apr 2021 19:05:33 GMT</pubDate>
            <atom:updated>2021-04-28T18:06:43.535Z</atom:updated>
            <content:encoded><![CDATA[<h3>Kafka Security Fundamentals— The Rosetta Stone to your event-streaming infrastructure</h3><h3>Getting Started</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T3C2pguEbESWXte6UDefcA.png" /><figcaption>Apache Kafka project official logo</figcaption></figure><p>Understanding <a href="https://kafka.apache.org/"><strong>Apache Kafka</strong></a> security is much like discovering the Rosetta Stone to the entire event-streaming landscape. This is not meant to be hyperbole at all, since once you understand a few common core configurations, you have the ability to connect any Kafka-based, event-driven application to dozens of Kafka offerings with relatively few changes. The vast majority of these changes come in the form of Kafka broker endpoints, as well as authentication configuration information in the form of protocol selection and providing credentials in some fashion. This blog post will cover the six most important Kafka security configuration options from a Kafka client perspective <em>(based on </em><a href="https://ibm-cloud-architecture.github.io/refarch-eda/"><strong><em>my team’s experiences</em></strong></a><em> so far)</em> to allow you to become as flexible as a Swiss Army Knife when working with any Kafka offering.</p><p>As this is a deeper dive into Apache Kafka and the capabilities it provides, I will not be covering any introductory Kafka material here. You can head over to <a href="https://kafka.apache.org/"><strong>kafka.apache.org</strong></a> for all the foundational information necessary or check out my friend Whitney’s <a href="https://www.youtube.com/watch?v=aj9CDZm0Glc&amp;t=1s"><strong>What is Kafka?</strong></a> video <em>(embedded below) </em>to learn everything you need to get started in under 10 minutes. Once you’ve read through everything here, if you still have questions or want to dig in further, the <a href="http://kafka.apache.org/documentation/#security"><strong>Security</strong></a> section of the official Kafka documentation covers everything in depth with a great deal of clarity — so that would be my first recommendation if you read through here to find that none of the common configurations below match your expected environment setups!</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Faj9CDZm0Glc%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Daj9CDZm0Glc&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Faj9CDZm0Glc%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/a569b91e6a70d5d7363bff70bdc95817/href">https://medium.com/media/a569b91e6a70d5d7363bff70bdc95817/href</a></iframe><p>Before we jump in, there are two caveats to address on the scope of this blog series:</p><ol><li>There are many, many ways to configure Kafka security from a Kafka administrator point-of-view. The focus of this short blog series is to more effectively comprehend Kafka security as a Kafka client (producer or consumer) and not a Kafka administrator (operations / provider). For a deeper understanding of practical Kafka operational security from an administrator’s point of view <em>(beyond the aforementioned </em><a href="http://kafka.apache.org/documentation/#security"><strong><em>Security</em></strong></a><em> docs above)</em>, you can reference Confluent’s <a href="https://docs.confluent.io/platform/current/security/general-overview.html"><strong>Security Overview</strong></a> for a much deeper understanding on how on to securely deploy and maintain Kafka well into the future!</li><li>There are two different patterns for authenticating with Kafka. The one we will address in this first post focuses on username and password-based authentication mechanisms. The other, which we will cover in a follow-up post, focuses on using SSL certificates for authentication, above and beyond their use in traditional encryption mechanisms. This latter pattern is traditionally referred to as mutual TLS-based authentication or mTLS, for short. If your current Kafka scenario doesn’t provide you with user-based credentials, don’t worry, we’ll cover that soon. The majority of this blog post will still hold true, with a few changes to the core six configuration properties covered below.</li></ol><p>Without any further ado, let’s get started!</p><h3>Kafka Settings</h3><h4><a href="http://kafka.apache.org/documentation/#adminclientconfigs_bootstrap.servers"><strong>bootstrap.servers</strong></a></h4><p>This is the most important setting as it tells Kafka clients where to look to talk to your applications. Sometimes this will be a string that contains just a single hostname:port combination, while other times it will be a comma-separated list of hostname:port combinations. Depending upon the Kafka offering you are connecting to, as well as how and where it is deployed, you will either be providing a singular bootstrap address or the entire collection of Kafka brokers in the same string. The bootstrap endpoint is a functional intermediary that Kafka offerings provide to allow for simpler networking in cases of expected fluctuations in the number of Kafka brokers, as it is a single URL that will return all the active brokers in the cluster to your clients automatically.</p><h4><a href="http://kafka.apache.org/documentation/#adminclientconfigs_security.protocol"><strong>security.protocol</strong></a></h4><p>This is the next most important security setting as it tells the clients how to communicate with the brokers. If you’ve ever had HTTPS issues while using a web browser, this is the setting that would give Kafka clients SSL issues while attempting to connect to Kafka brokers! If you are receiving errors in your application while attempting to connect to Kafka with references to a broker(-1), you will usually need to revisit this setting <em>(i.e. you should be using </em><em>SASL_SSL but you configured </em><em>PLAINTEXT, etc.)</em>.</p><p>The valid values are:</p><ul><li>PLAINTEXT <em>(using PLAINTEXT transport layer &amp; no authentication — default value)</em></li><li>SSL<em> (using SSL transport layer &amp; certificate-based authentication)</em></li><li>SASL_PLAINTEXT <em>(using PLAINTEXT transport layer &amp; SASL-based authentication)</em></li><li>SASL_SSL <em>(using SSL transport layer &amp; SASL-based authentication)</em></li></ul><h4><a href="http://kafka.apache.org/documentation/#adminclientconfigs_ssl.truststore.location">ssl.truststore.location</a></h4><p>To allow for encryption of communication between Kafka brokers and clients, as specified by our security.protocol setting above, we need to provide our Kafka clients with the location of a trusted Certificate Authority-based certificate. This file is often provided by the Kafka administrator and is generally unique to the specific Kafka cluster deployment. It is outside the scope of this blog post to determine how you provide access to this file for your individual runtimes, but it must exist inside the runtime container somewhere.</p><p>For Java-based clients, this file needs to be in the JKS format. For Python or NodeJS-based clients, this file should be in either a PEM or P12 format. To extract a PEM-based certificate from a JKS-based truststore, you can use the following command:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/950748b9d9a3a023b40454d122af7b01/href">https://medium.com/media/950748b9d9a3a023b40454d122af7b01/href</a></iframe><h4><a href="http://kafka.apache.org/documentation/#adminclientconfigs_ssl.truststore.password">ssl.truststore.password</a></h4><p>When providing a JKS-based truststore for validation of encryption certificates via the ssl.truststore.location configuration property above, you will generally need to provide a password to access the truststore file. This should be provided by your Kafka administrator at the same time they provide you with the truststore <em>(or at least provide you with a method to acquire the truststore password)</em>. Truststore passwords are not required for PEM-based truststores, only JKS-based truststores.</p><h4><a href="http://kafka.apache.org/documentation/#adminclientconfigs_sasl.mechanism">sasl.mechanism</a></h4><p>SASL stands for <a href="https://en.wikipedia.org/wiki/Simple_Authentication_and_Security_Layer"><em>Simple Authentication and Security Layer</em></a> and focuses on decoupling authentication mechanisms from application protocols — something that the majority of the early Internet-era applications and middleware did a very poor job of. As such, there are a number of pluggable mechanisms that can be included here to determine authentication between Kafka clients and brokers once the necessary communication protocol has been established with our previous setting.</p><p>The most commonly encountered values here are:</p><ul><li>PLAIN <em>(cleartext passwords, although they will be encrypted across the wire per </em><em>security.protocol settings above)</em></li><li>SCRAM-SHA-512 <em>(modern Salted Challenge Response Authentication Mechanism)</em></li><li>GSSAPI <em>(Kerberos-supported authentication and the default if not specified otherwise)</em></li></ul><h4><a href="http://kafka.apache.org/documentation/#adminclientconfigs_sasl.jaas.config">sasl.jaas.config</a></h4><p>Java-based clients make use of the <a href="https://docs.oracle.com/javase/8/docs/technotes/guides/security/jaas/JAASRefGuide.html">Java Authentication and Authorization Service or JAAS</a> framework that extends the plug-ability of the SASL decoupling approach to the enterprise-secured Java programming language. This allows for flexible yet robust security configuration through JAAS configuration. All of this boils down to you needing to provide a string named sasl.jaas.config that contains three main components: (1) a class name, (2) a username, and (3) a password. The security components and plugins take it from there. This string is often provided by your Kafka administrator with all the components already filled in for you.</p><p>Some commonly encountered sasl.jaas.config strings are:</p><ul><li>org.apache.kafka.common.security.plain.PlainLoginModule required username=&quot;{USERNAME}&quot; password=&quot;{PASSWORD}&quot;;</li><li>org.apache.kafka.common.security.scram.ScramLoginModule required username=&quot;{USERNAME}&quot; password=&quot;{PASSWORD}&quot;;</li></ul><h3>Common configuration options for the most popular offerings</h3><h4>Strimzi / Red Hat AMQ Streams / IBM Event Streams</h4><p><a href="https://strimzi.io/"><strong>Strimzi</strong></a> is an open-source project which provides a Kubernetes Operator-focused way of deploying Apache Kafka clusters flexibly, repeatedly, and securely to any Kubernetes distribution. The enterprise-grade offerings of <a href="https://www.redhat.com/en/blog/getting-started-red-hat-amq-streams-operator"><strong>Red Hat AMQ Streams</strong></a> and <a href="https://ibm.github.io/event-streams/"><strong>IBM Event Streams</strong></a> are built on top of the underlying Strimzi Operator Custom Resources and take advantage of much of the same configuration specifications.</p><p>Clients external to the Kubernetes or OpenShift cluster which Strimzi et al are deployed to often use the following configurations:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cf413583ec9d62606b34105c6a117234/href">https://medium.com/media/cf413583ec9d62606b34105c6a117234/href</a></iframe><p>Clients internal to the Kubernetes or OpenShift cluster which Strimzi et al are deployed to often use the following configurations: <em>(note the difference as these clients do not require SSL-based encryption by default as they are local to the cluster)</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/96c24174f07273852f656480d86e40ef/href">https://medium.com/media/96c24174f07273852f656480d86e40ef/href</a></iframe><blockquote><strong>NOTE:</strong> Recent Strimzi versions have added support for OAuth-based authentication, beyond just SASL or mTLS-based authentication. We have not widely encountered this in the field yet, but will address it once we do.</blockquote><p>For more information on the specifics of configuring <a href="https://strimzi.io/docs/operators/latest/using.html#assembly-securing-access-str"><strong>secure listeners on Strimzi-based Kafka clusters</strong></a>, you will want to visit the official <a href="https://strimzi.io/docs/operators/latest/using.html"><strong>Strimzi Docs</strong></a>. The examples in the documentation, as well as our demo scenario below, utilize three listeners with different security profiles <em>(PLAINTEXT vs SSL transport layers / no authentication vs SASL-based authentication vs mutual TLS authentication / internal vs external clients)</em> based upon the expected clients connecting to them.</p><p>For information on configuring users and access credentials, visit the <a href="https://strimzi.io/docs/operators/latest/using.html"><strong>Strimzi Docs</strong></a> page for <a href="https://strimzi.io/docs/operators/latest/using.html#con-securing-client-authentication-str"><strong>User Authentication</strong></a> as the least-common-denominator between all three noted Strimzi variants. This method of user administration leverages the User Operator provided by Strimzi to manage users through Kubernetes Custom Resources.</p><h4>Confluent Platform</h4><p>Confluent provides many configuration options and deployment targets for their Confluent Platform on-premises Kafka offering. Our team’s main focus is Red Hat OpenShift-based deployments and as such, we focus on the capabilities made available to Confluent Platform through the <a href="https://docs.confluent.io/operator/current/overview.html"><strong>Confluent Operator</strong></a><strong> </strong>and<strong> </strong><a href="https://docs.confluent.io/operator/current/co-authenticate.html"><strong>SASL and mTLS-based authentication mechanisms with optional TLS-encrypted traffic</strong></a>.</p><p>Clients external to the Kubernetes or OpenShift cluster which Confluent Platform is deployed to most often use the following configurations:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/52eef50f4eb5e9cc12e02146e5479786/href">https://medium.com/media/52eef50f4eb5e9cc12e02146e5479786/href</a></iframe><p>Clients internal to the Kubernetes or OpenShift cluster which Confluent Platform is deployed to most often use the following configurations:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fbc62f404ef4876a87681f4094480733/href">https://medium.com/media/fbc62f404ef4876a87681f4094480733/href</a></iframe><p>For information on configuring custom SASL users with the Confluent Operator, visit the <a href="https://docs.confluent.io/operator/current/overview.html"><strong>Confluent Docs</strong></a> page for <a href="https://docs.confluent.io/operator/current/co-authenticate.html#add-custom-sasl-users"><strong>Authentication and Encryption</strong></a><strong>.</strong></p><p><strong>IBM Event Streams on Cloud</strong></p><p>IBM provides a hosted, SaaS-based Kafka offering as part of IBM Cloud and it is available via the <a href="https://cloud.ibm.com/catalog/services/event-streams"><strong>IBM Cloud Catalog</strong></a>. It comes pre-configured with everything you need to immediately get started working with Kafka applications, along with authentication that is directly integrated into the IBM Cloud IAM.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/040c3da66b188ca1abc58b457c84b99f/href">https://medium.com/media/040c3da66b188ca1abc58b457c84b99f/href</a></iframe><p>As the IBM Event Streams on Cloud does not leverage a bootstrap address for client connectivity, the bootstrap.servers property is a list of broker endpoint addresses instead of a single broker bootstrap proxy endpoint.</p><p>For information on configuring users and access credentials, visit the <a href="https://cloud.ibm.com/docs"><strong>IBM Cloud Docs</strong></a> page for <a href="https://cloud.ibm.com/docs/EventStreams?topic=EventStreams-connecting"><strong>Connecting to Event Streams</strong></a> and <a href="https://cloud.ibm.com/docs/EventStreams?topic=EventStreams-kafka_using#kafka_clients"><strong>Kafka Clients</strong></a>.</p><h3>Practical Use Case with Kafka binaries</h3><p>Now we will walk through a straight-forward use case of deploying a vanilla open-source Strimzi cluster <em>(which remember uses the same underlying configuration as IBM Event Streams V10 and Red Hat AMQ Streams)</em>, creating a SCRAM-based user, creating a topic, and interacting with that topic through the CLI tools provided by the Kafka binaries.</p><p>The only parameters we will be configuring from the downstream client perspective are the ones we covered above. This quick tutorial assumes access to a Red Hat OpenShift 4.5+ cluster. If you do not have one, you can access one easily via the <a href="https://developer.ibm.com/openlabs/openshift"><strong>IBM Open Labs</strong></a>.</p><p><strong><em>NOTE:</em></strong><em> You will want to avoid the OpenShift Playground clusters available on Katacoda for this exercise, as they use a more complex SSL certificate setup than the scope of this article is meant to address.</em></p><h4>Scenario setup</h4><ul><li>Access an OpenShift 4.5+ cluster via the OpenShift Console.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vU5VJL1ovp6PuWNdovMieQ.png" /></figure><ul><li>Go to Operators on the left hand side menu and click <strong>OperatorHub</strong>. Then, search for <em>strimzi</em>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BKYo4scFSV6dNrpRoam8VQ.png" /></figure><ul><li>Click the <strong>Strimzi</strong> tile and click on Install.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZnI10Q5BtYhEYcOMzAk9ig.png" /></figure><ul><li>Make sure you are installing from the <strong><em>stable</em></strong> channel and you are installing the operator in <strong><em>All namespaces on the cluster</em></strong>. Click <strong>Install</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PNoNoqZ9ZmxJT5WyhWIi-g.png" /></figure><ul><li>After a couple of minutes, you should see the <strong>Strimzi</strong> operator successfully deployed and running.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i3YEBFejw4S09OL2CAcxGA.png" /></figure><ul><li>Go to <strong>Projects</strong> on the left-hand side menu and click <strong>Create Project</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tn9z1s0xck1J8gewTjm5Ww.png" /></figure><ul><li>Once the project is created, you should be presented with a dashboard for it.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*24OsejpIGD0Eubk9N_3meA.png" /></figure><h4>Create a Kafka cluster instance</h4><ul><li>Click <strong>Installed Operators</strong> under the Operators section in the left hand side menu. Click the <strong>Strimzi</strong> operator.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FMPIMEmVOfLXxRARRuWz0w.png" /></figure><ul><li>Click the <strong>Kafka</strong> tab and click <strong>Create Kafka</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fL0DTk7CYl_LKVjHL_oWIQ.png" /></figure><ul><li>Switch to the YAML view.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5chrUTkixd57SAp0GFcnYA.png" /></figure><ul><li>Update the listeners section within the .spec section with the following configuration:</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5e0b18e8aa86ca01a8b157a4dd19c2b7/href">https://medium.com/media/5e0b18e8aa86ca01a8b157a4dd19c2b7/href</a></iframe><p>The above <em>listeners</em> configuration will create an internal non-secured listener on port 9092, an internal TLS-secured listener on port 9093 and an external TLS-secured and SCRAM-SHA-512-authentication-required on port 9094. We will use this last port to access our Kafka cluster from outside of OpenShift. As a result, we will need a TLS certificate for the SSL connection plus a set of SCRAM-SHA-512 credentials to get authenticate with.</p><ul><li>Make sure the entityOperator along with the topicOperator and the userOperator are also defined within the spec section. If they are not, add them.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aMlyZOYoCxEv3GgAcJUr-A.png" /></figure><ul><li>Once you have updated the listeners section, click <strong>Create</strong>. After a minute or so, you should see the Kafka Cluster ready.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m34mRkH_q1SLFi0PqA5ouQ.png" /></figure><h4>Create a Kafka topic via the Strimzi Operator.</h4><ul><li>Click the <strong>KafkaTopic</strong> tab and then click <strong>Create KafkaTopic:</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XQIUTEoyKiK0aqvNSmXLFw.png" /></figure><ul><li>Leave the defaults and click <strong>Create</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LYNV3gpMuHJlQv755HNA2w.png" /></figure><blockquote><strong>NOTE:</strong> Your Topic Name should stay my-topic for this simple example to align with the default ACLs for the KafkaUser you will create next.</blockquote><ul><li>You should see your topic created.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MZKbnKMZNo4p5fgeaAlw0g.png" /></figure><h4>Create a KafkaUser via the Strimzi Operator</h4><ul><li>Click the <strong>KafkaUser</strong> tab and then click <strong>Create KafkaUser</strong>:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1h97Wl3QlJijqpI0plNHlg.png" /></figure><ul><li>Click <strong>Authentication</strong> and switch the authentication type to scram-sha-512. Click <strong>Create</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iUAuj8sVLsemubbOOandJA.png" /></figure><p>This will create a Kafka user with a set of scram-sha-512 credentials so that it can access the Kafka Cluster from outside of OpenShift using the external listener on port 9094 you created earlier.</p><ul><li>You should see the user created.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gb58RT7GpvO-72CguijDOw.png" /></figure><h4>Generate configuration properties</h4><p>On your terminal, execute the following commands:</p><ul><li>Download the Kafka binaries to your sandbox environment.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0601d26ed1f15615c29e22f5c16788a9/href">https://medium.com/media/0601d26ed1f15615c29e22f5c16788a9/href</a></iframe><ul><li>Change project to kafka-security.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8b8ff4e070b8ec36ee9c052848a80e41/href">https://medium.com/media/8b8ff4e070b8ec36ee9c052848a80e41/href</a></iframe><ul><li>Export our configuration data to environment variables for easy reuse.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8f21cc70b019aaac5ee372105b44937e/href">https://medium.com/media/8f21cc70b019aaac5ee372105b44937e/href</a></iframe><ul><li>Setup our three basic SASL/SSL properties.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/df58f5d1c5550e1c56401b89fc46693f/href">https://medium.com/media/df58f5d1c5550e1c56401b89fc46693f/href</a></iframe><p>Since we are going to work with the Kafka cluster we deployed earlier from outside of the OpenShift cluster, we need to do that through the external SSL-secured and SCRAM-SHA-512-authentication-protected 9094 Kafka listener we defined earlier <em>(the Strimzi operator will take care of creating the external access to that listener within the OpenShift cluster in the form of an OpenShift Route)</em>.</p><p>As a result, the security.protocol we need to set in the configuration file that the Kafka binaries scripts will later use is SASL_SSL. Then, we need to provide two things for this type of connection to a Kafka cluster. One is the SSL certificate for the secure communication and the other is the SCRAM-SHA-512 credentials. The first is provided through the ssl.truststore.* properties acquired in the next two steps. The other, the SCRAM-SHA-512 credentials, are provided through the sasl.* properties above.</p><ul><li>Play the role of the Kafka Administrator and generate your own Java Truststore artifact. This is the certificate that will allow us to establish the SSL communication to our Kafka cluster.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9fb2e0161037abce6805b7a593129278/href">https://medium.com/media/9fb2e0161037abce6805b7a593129278/href</a></iframe><ul><li>Pass in the path to the Java Truststore artifact on the local filesystem (with absolute path).</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/12378032de3732a21123ab800cdbb306/href">https://medium.com/media/12378032de3732a21123ab800cdbb306/href</a></iframe><h4>Run the Kafka binaries</h4><p>On your terminal, execute the following commands to validate our connection properties and interact with the Kafka brokers:</p><ul><li>View topic data via <em>bin/kafka-topic.sh</em> tool:</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/90b3426e937e72a3c877d7bf47fa7cd7/href">https://medium.com/media/90b3426e937e72a3c877d7bf47fa7cd7/href</a></iframe><ul><li>Send data with the <em>bin/kafka-console-producer.sh</em> tool:</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f87b6c46da010e8fe439676f1ce0a53c/href">https://medium.com/media/f87b6c46da010e8fe439676f1ce0a53c/href</a></iframe><blockquote><strong>NOTE:</strong> Use `Ctrl+C` when you are done sending data to exit the console producer.</blockquote><ul><li>Read data with the <em>bin/kafka-console-consumer.sh</em> tool:</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a7d6376cec3875a848f0b1e1e5764090/href">https://medium.com/media/a7d6376cec3875a848f0b1e1e5764090/href</a></iframe><blockquote><strong>NOTE:</strong> Use `Ctrl+C` when you are done reading data to exit the console consumer.</blockquote><p>With this understanding of Kafka client security configuration, you can now seamlessly switch between Strimzi to IBM Event Streams to Red Hat AMQ Streams to Confluent Platform to IBM Event Streams on Cloud to <a href="https://lenses.io/"><strong>Lenses.io</strong></a> to <a href="https://github.com/obsidiandynamics/kafdrop"><strong>kafdrop</strong></a> to <a href="https://github.com/edenhill/kafkacat"><strong>kafkacat</strong></a> to <a href="https://github.com/apache/kafka/tree/trunk/bin"><strong>Kafka-provided binaries</strong></a> and beyond by following the same process!</p><h3>Conclusion</h3><p>These same six settings will occur over and over again, from Producer Configs to Consumer Configs to Kafka Streams Configs to Kafka Connect Configs to Admin Client Configs to command-line configs. They are everywhere! Depending on your implementation language, platform, and tooling, there may be some assistance available in the SDKs, APIs, or CLIs to prevent the need of copying and pasting across the different Kafka component configurations, but once you get the configuration correct at the highest level, everything else in your event-driven applications seems much simpler to connect, debug, and validate at the lower levels!</p><p>I hope this whirlwind tour of practical Kafka security configuration options has been helpful and will become a handy reference for you to use going forward! Please let me know if you find this useful and would like to see more practical Kafka experiences documented here. In the meantime, head over to our <a href="https://www.ibm.com/cloud/architecture/architectures/eventDrivenArchitecture/overview"><strong>Event-Driven Reference Architecture</strong></a> in the <a href="https://www.ibm.com/cloud/architecture/architectures"><strong>IBM Cloud Architecture Center</strong></a> that has the latest and greatest collection of all things event-driven and Kafka-based!</p><p><em>A special shoutout of thanks to </em><a href="https://www.linkedin.com/in/jesus-almaraz-hernandez/"><em>Jesus Almaraz</em></a><em> &amp; </em><a href="https://www.linkedin.com/in/jeromeboyer/"><em>Jerome Boyer</em></a><em> for their reviews of, edits on, and polishing contributions to this article.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=518f49640db4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Migrating Java microservices apps from Spring Boot to MicroProfile]]></title>
            <link>https://medium.com/ibm-cloud/migrating-java-microservices-from-spring-boot-to-microprofile-ibm-cloud-blog-a855e4b03624?source=rss-5adc9417d038------2</link>
            <guid isPermaLink="false">https://medium.com/p/a855e4b03624</guid>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[spring-boot]]></category>
            <dc:creator><![CDATA[Rick Osowski]]></dc:creator>
            <pubDate>Mon, 24 Sep 2018 14:00:08 GMT</pubDate>
            <atom:updated>2018-10-09T17:16:05.116Z</atom:updated>
            <content:encoded><![CDATA[<h3>Migrating Java microservices app from Spring Boot to MicroProfile</h3><p>So what do you do once you have a hypothesis and some time on your hands? You perform an experiment! And that’s just what our Solution Engineering team did.</p><p>In this blog series, you will see the work done by our team to convert the existing Spring-based implementation of our <a href="https://www.ibm.com/support/knowledgecenter/en/SSAW57_liberty/com.ibm.websphere.wlp.nd.multiplatform.doc/ae/cwlp_about.html">WebSphere Liberty-based</a> BlueCompute application to using the latest and greatest of the generally available MicroProfile 1.3 project capabilities. Our BlueCompute application is a microservices-based reference architecture that provides a digital storefront experience to users while being built with containers and deployed on Kubernetes.</p><p>This series will provide insight into the well-worn and edge cases when implementing standard microservices projects in either framework. Spring or MicroProfile, which one should you pick for more interaction between backend microservices? Which one is more easily adaptable to different user interaction patterns? And, most of all, which one allows you to write less code, while still understanding the breadth of the application?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/0*0ucy2LuN2qI3RBJZ.png" /></figure><p>If you’re writing Java-based microservices today, you won’t want to miss this series. It will show you the details about what it’s like to build, deploy, and migrate Java microservices with Spring or MicroProfile, two of the most popular frameworks out there today.</p><p>If you would like more background on Java microservices architectures, please check out the following resources:</p><p>You can also explore the <a href="https://openliberty.io/guides/">OpenLiberty guides</a> to become a MicroProfile master.</p><h3>The story behind our migration from Spring to MicroProfile</h3><p>In recent years, Enterprise Java has been defined by two major benefactors — Spring on one end of the spectrum and Java Enterprise Edition on the other end. As with all competing, evolving, or somewhat overlapping functional areas, developers always argue which one is “better” or which one is “best.”</p><p>Is it better to future-proof yourself by following a widely accepted community development model, as defined by the Java Community Process and brought us all the Java EE specifications? Or is it better to adopt smaller components that are more applicable on a per-project basis but may wildly evolve in future releases as the market shifts? Can I adopt the latest and greatest architectural conventions, like microservices and containers, with a framework that is slow to move and adverse to picking winners? Or do I choose a framework that will adapt quickly, but cause me to rewrite much of my application code to stay current as the winners evolve? Good questions!</p><h3>Lessons learned while writing our reference microservices-based app</h3><p>At this point, it’s fair to ask: Why present this history lesson of MicroProfile?</p><p>Simply put, our Solution Engineering team was in this very situation and had to decide which way to go. Our Solution Engineering team builds reference applications and architectures for the entire spectrum of Enterprise Workloads, including microservices, cloud-native development, messaging and integration-based workloads, batch workloads, mainframe workloads, and so on.</p><p>Not shortly after the original release of the MicroProfile 1.0 specification, our team was tasked with building a reference application that would become our exemplar for the burgeoning microservices architectural space. What would soon become known as “BlueCompute” would have to be architected from scratch, on whiteboards and in conference rooms, making sure it hit all the right points necessary to truly represent a proper reference architecture. As mentioned above, MicroProfile had only recently been released to the general public (if not still in preview mode), so it wasn’t something the team felt strongly about adopting, as our team didn’t know how broadly we could implement a proper end-to-end, microservices-based, 12-factor-oriented application with “just MicroProfile.”</p><p>So, for lack of other options, the team decided to implement the initial version of our BlueCompute reference application utilizing the Spring Boot and Spring Cloud frameworks. Development was rather straightforward, with some minor hiccups due to supported features in specific versions of either the Spring Boot or Spring Cloud sub-projects, but our team was able to achieve exactly what we needed to for a full reference implementation on both a Cloud Foundry platform and a Docker-based container workload.</p><p>The next wrinkle was the success of Kubernetes in the container orchestration workspace. So, instead of managing many containers locally or through a series of scripts, our team now had the option to use one CLI to deploy and scale our reference application to (resource and budget notwithstanding) near-infinite technical bounds. However, the move to Kubernetes mostly obfuscated the need for Spring Cloud capabilities, as most of our microservice capabilities were supported out of the box by Kubernetes (with some minor exceptions like Hystrix fault tolerance support).</p><h3>Reduce complexity and dependencies of Java microservices architectures</h3><p>If you’re interested in reducing the amount of complexity and dependencies in your Java microservices architectures, then you’ll want to follow along with this multi-part series. It will cover specific details of the MicroProfile framework and how our team migrated from an original Spring Boot-based implementation to one built on top of the latest MicroProfile packaging.</p><p>Additionally, if you would like some hands-on experience, check out <a href="https://openliberty.io/guides/">guides on openliberty.io</a>. There are over a dozen MicroProfile-specific tutorials for deploying Java applications on the world’s most flexible, open-source application runtime, built for cloud native and microservices!</p><p><em>Originally published at </em><a href="https://www.ibm.com/blogs/bluemix/2018/09/migrate-java-microservices-from-spring-to-microprofile-p1/"><em>www.ibm.com</em></a><em> on September 24, 2018.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a855e4b03624" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ibm-cloud/migrating-java-microservices-from-spring-boot-to-microprofile-ibm-cloud-blog-a855e4b03624">Migrating Java microservices apps from Spring Boot to MicroProfile</a> was originally published in <a href="https://medium.com/ibm-cloud">IBM Cloud</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>