<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jeyakumaran Chandrasegaram [JK] on Medium]]></title>
        <description><![CDATA[Stories by Jeyakumaran Chandrasegaram [JK] on Medium]]></description>
        <link>https://medium.com/@jkumaranc?source=rss-4f7bc551734e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Jcbc6ZUxX4xICmGxq6v-cg.png</url>
            <title>Stories by Jeyakumaran Chandrasegaram [JK] on Medium</title>
            <link>https://medium.com/@jkumaranc?source=rss-4f7bc551734e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 14:36:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jkumaranc/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Monty Hall Paradox in Java]]></title>
            <link>https://medium.com/@jkumaranc/monty-hall-paradox-in-java-115602e31c0b?source=rss-4f7bc551734e------2</link>
            <guid isPermaLink="false">https://medium.com/p/115602e31c0b</guid>
            <category><![CDATA[java]]></category>
            <category><![CDATA[monty-hall]]></category>
            <category><![CDATA[monty-hall-problem]]></category>
            <dc:creator><![CDATA[Jeyakumaran Chandrasegaram [JK]]]></dc:creator>
            <pubDate>Tue, 26 Jul 2022 00:39:45 GMT</pubDate>
            <atom:updated>2022-07-26T00:39:45.162Z</atom:updated>
            <content:encoded><![CDATA[<pre>import java.util.HashMap;</pre><pre>import java.util.Map;</pre><pre>import java.util.Random;</pre><pre>public class GoatAndCarProblem</pre><pre>{</pre><pre>public static void main(String[] args)</pre><pre>{</pre><pre>int numberOfTimesCarChosenIfSwitched = 0;</pre><pre>int numberOfTimesGamePlayed = 10000;</pre><pre>for (int j = 0; j &lt; numberOfTimesGamePlayed; j++)</pre><pre>{</pre><pre>Map doors = new HashMap();</pre><pre>doors.put(1, &quot;Goat&quot;);</pre><pre>doors.put(2, &quot;Goat&quot;);</pre><pre>doors.put(3, &quot;Car&quot;);</pre><pre>//you select a door number randomly</pre><pre>final int doorNoOfYourOriginalChoice = new Random().nextInt(doors.size());</pre><pre>eliminateDoorWithGoat(doors, doorNoOfYourOriginalChoice);</pre><pre>String switchedChoice = switchYourChoice(doors, doorNoOfYourOriginalChoice);</pre><pre>//Check whether the switched choise is the door with car.</pre><pre>if (switchedChoice.equals(&quot;Car&quot;))</pre><pre>{</pre><pre>numberOfTimesCarChosenIfSwitched++;</pre><pre>}</pre><pre>}</pre><pre>//Calculate the probability.</pre><pre>System.out.println(&quot;The chance of car been picked if switched &quot; + ((float)numberOfTimesCarChosenIfSwitched/(float)numberOfTimesGamePlayed)*100);</pre><pre>}</pre><pre>private static void eliminateDoorWithGoat(Map doors, final int doorNoOfYourChoice)</pre><pre>{</pre><pre>//now host comes and eliminate one of the doors with goat, which you haven&#39;t picked.</pre><pre>for (Map.Entry door : doors.entrySet())</pre><pre>{</pre><pre>if (door.getKey() != doorNoOfYourChoice // cannot eliminate the door you chosen.</pre><pre>&amp;&amp; door.getValue().equals(&quot;Goat&quot;))</pre><pre>{</pre><pre>//The door with goat which is now eliminated by the host.</pre><pre>doors.remove(door.getKey());</pre><pre>break;</pre><pre>}</pre><pre>}</pre><pre>}</pre><pre>private static String switchYourChoice(Map doors, final int doorNoOfYourChoice)</pre><pre>{</pre><pre>//There will be only two doors left, and you switch to other door.</pre><pre>for (Map.Entry door : doors.entrySet())</pre><pre>{</pre><pre>if (door.getKey() != doorNoOfYourChoice)</pre><pre>{</pre><pre>return door.getValue();</pre><pre>}</pre><pre>}</pre><pre>return null;</pre><pre>}</pre><pre>}</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=115602e31c0b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Pro’s and Con’s of using AWS MSK Serverless — DigIO]]></title>
            <link>https://medium.com/digio-australia/the-pros-and-con-s-of-using-aws-msk-serverless-digio-ce4875de2296?source=rss-4f7bc551734e------2</link>
            <guid isPermaLink="false">https://medium.com/p/ce4875de2296</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[msk]]></category>
            <category><![CDATA[serverless]]></category>
            <dc:creator><![CDATA[Jeyakumaran Chandrasegaram [JK]]]></dc:creator>
            <pubDate>Mon, 18 Jul 2022 04:39:43 GMT</pubDate>
            <atom:updated>2022-08-04T04:20:03.611Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Pro’s and Con’s of using AWS MSK Serverless — DigIO</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*oTpbIAflIx3auj0k.png" /></figure><p>Author: Sinnappu Jeyakumaran<br><em>Senior Software Engineer at DigIO</em></p><p>Our team recently did a tech assessment on AWS MSK Serverless, which automatically scales compute and storage resources so you can stream and store Apache Kafka data on demand.</p><p>Back in 2018, Amazon announced its own version of managed streaming for Kafka (MSK). Amazon MSK offered an alternative to running self hosted Kafka with fully user managed infrastructure. Amazon MSK (now referred to as ‘provisioned’ to avoid confusion with its serverless addition) is a fully managed kafka service which means one doesn’t have to worry about server provisionings, upgrades, Zookeeper setups and all the relevant coordinating tasks. It rapidly reduces the operational burden of setup and deployment. It is automatically placed inside Amazon VPC. The brokers can be spread across different availability zones. You can use the Amazon Glue schema registry comprehensively with MSK.</p><p>No doubt, ‘MSK Provisioned’ nicely blends with the rest of the amazon ecosystem. It enables AWS users to add their Kafka use cases more efficiently. But its limitations would start to uncover as we continue to use it. With “MSK Provisioned”, we still have to manually monitor to handle scalability. If the demand is fluctuating, users may have to manually scale up and down the capacity. With that, they have to redo their load balancing with the current set of brokers. If the demand variation has a pattern of its own, then users may have to find a way to automate this scaling process by themselves. Otherwise they would have to pay many hundreds to thousands dollars for unnecessary idle time. And let’s not forget the occasional spikes. Users may face delays, connection errors or long connection times if those spikes are not handled properly. This operational and maintenance cost and effort could be a big burden, especially if you are a small team working on a relatively small greenfield initiative.</p><p>“MSK Serverless” solves this problem to some extent. This ‘serverless’ cluster type offering of MSK scales up and down automatically. It maintains and scales the brokers internally and we don’t even get to see the details. All we get is a single bootstrap url string. “MSK Serverless” automatically places the partitions to new brokers and rearranges the placements when brokers scale down. It is that simple to use.</p><p>Lets look into the pros and cons of “MSK Serverless” cluster.</p><h3>Pro’s of AWS MSK Serverless</h3><h4>1. Automatic Capacity Management</h4><p>Capacity in serverless clusters are automatically managed. Brokers are internally created and managed. Users won’t have to compute and create the number of brokers. The partitions would also be automatically placed as when brokers internally scaled either way.</p><h4>2. Simple to use</h4><p>Like MSK Provisioned, Serverless is also very easy to use. You can create serverless clusters with a matter of a few clicks and it takes not more than five minutes to spin up. We can assign the VPC and security groups along the way and once done the service is up and running. Deleting a cluster is equally easy too.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pvKVTzxZFNmSzbB8.png" /></figure><h4>3. Amazon Glue for Schema Registry</h4><p>We can use the fully managed Amazon Glue as the schema registry for MSK clusters. The integration is easy and straightforward. Glue supports all three (JSON, Protobuf, AVRO) schema types. Worthy to mention is, Glue registry has public access enabled, so it is still possible to use the same Glue schema registry for local development with dockerised standalone Kafka instance.</p><h3>Con’s of AWS MSK Serverless</h3><h4>1. Lack of control</h4><p>As one would expect, serverless comes with limited control compared to the ‘provisioned’ type. We can’t control the number of brokers or which Kafka version to be used or system operations like reboots. At the time of writing this article, we are able to create serverless clusters on AWS web console only.</p><p>CDK and CloudFormation are not supported yet for serverless cluster type. Although a feature <a href="https://github.com/aws-cloudformation/cloudformation-coverage-roadmap/issues/1209">request</a> has been raised and this may get resolved in due time.</p><h4>2. No public access</h4><p>Unlike the “provisioned” cluster type, serverless cluster doesn’t offer any public access to Kafka service. Only the AWS services which are under the configured VPCs (up to five) allowed access to the serverless cluster. This means, the external applications which want to connect to the cluster will have to do this via an AWS bastion proxy. Another option is to use the open source Kafka Rest Proxy by Confluent. This also means any local development environment may have to be content with local dockerised Kafka instances (or use a bastion proxy).</p><h4>3. Lack of library support</h4><p>MSK Serverless (and Provisioned) use the SASL mechanism of ‘AWS_MSK_IAM’. This is well supported in Java (or any other JVM language) via aws-msk-iam-auth module. But if your services are using a non JVM language, you would most likely be the early adapter and have to do the hard yard of implementing these mechanisms. At the time of writing, there were no GoLang and Python libraries available to connect to MSK serverless using the ‘AWS_MSK_IAM’ mechanism. Although this might most likely change in the future.</p><h4>4. AWS Console is no “Confluent Control Center”</h4><p>As its competitors often claim, AWS Serverless is cloud hosted but not cloud native. The web console offers a very simple UI to perform very basic Kafka operations. It doesn’t have the complex features of the Confluent control center. You cannot manage the topics. You cannot do on-console KSQL stream processing. You cannot monitor the consumers and producers. The list keeps on growing.</p><h4>5. Not for high intensity large workloads</h4><p>Each serverless cluster has a maximum write throughput of up to 200 MiB per second and a maximum read throughput of up to 400 MiB per second. The maximum number of partitions allowed is 120 and data retention is maximum of 1 day. This could be a limitation for very large workloads and high intensity events based use cases.</p><h4>6. Cost does not scale to zero!</h4><p>Although the pricing for MSK Serverless is throughput based, it still charges for cluster hours. That means, even when the cluster is idle, we have to pay some amount.</p><p>Here’s the pricing breakdown for MSK Serverless (for Asia Pacific Sydney region). This scenario is sourced from the AWS pricing model.</p><p>Let’s assume the cluster has 5 topics with 20 partitions each. Daily, your producers write on average 100GB of data and your consumers read 200GB of data. You also retain that data for 24 hours to ensure it is available for replay. In the above scenario, you would pay the following for a 31-day month:</p><p>ChargesUsageRate (USD)Sub totalsCluster-hours31 days * 24 hrs/day = 744 cluster-hours0.9375/cluster-hr744 * 0.9375 = 697.5Partition-hours31 days * 24 hrs/day * 5 * 20 = 74,400 partition-hours0.001875/partition-hr74,400 * 0.001875 = 139.5Data-In100 GB x 31 days = 3,100 GB0.12/GB-in3,100 * 0.125 = 387.5Data-Out200 GB x 31 days = 6,200 GB0.05/GB-out6,200 * 0.0625 = 387.5StorageAverage storage used = 100 GB-months0.10/GB-month100 * 0.12 = 12Total1623 USD</p><p>As you can see it would cost at least 700 USD for a month long idle, inactive and empty serverless cluster. Any data and storage cost is proportional to the throughput and this could vary depending on the load.</p><h3>Conclusion</h3><h4>We recommend that MSK Serverless Kafka is best used for following scenarios:</h4><ul><li>For startups, small scale applications which use the Kafka platform.</li><li>For those services with unpredictable varying data load.</li><li>As of now, only JVM applications are best equipped with all the necessary libraries to use MSK Serverless.</li><li>Since the public access is limited, it is recommended for AWS ecosystem only while local development can best utilise the dockerised Kafka instances.</li><li>Amazon Glue registry is the preferred managed schema registry for MSK Serverless (and the same registry can be used with local development too).</li><li>Teams will have to provision the cluster manually in the AWS console for the time being. Lack of CloudFormation support can be a DevOps hurdle although this could be addressed by AWS duly.</li></ul><h3>Sample Application</h3><p>We have developed a small and simple Kotlin spring boot application which connects MSK serverless to produce and consume data for your reference. It also has samples for connecting to the Amazon Glue registry. For local ‘development’ profile, it uses dockerised Confluent Kafka instance.</p><p>Here’s the <a href="https://github.com/digio/kafka-msk-serverless-kotlin-tutorial">link</a> to the git repository for the project.</p><p><em>Originally published at </em><a href="https://digio.com.au/the-pros-and-cons-of-using-aws-msk-serverless/"><em>https://digio.com.au</em></a><em> on July 18, 2022.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ce4875de2296" width="1" height="1" alt=""><hr><p><a href="https://medium.com/digio-australia/the-pros-and-con-s-of-using-aws-msk-serverless-digio-ce4875de2296">The Pro’s and Con’s of using AWS MSK Serverless — DigIO</a> was originally published in <a href="https://medium.com/digio-australia">DigIO Australia</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>