<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Banula Kumarage on Medium]]></title>
        <description><![CDATA[Stories by Banula Kumarage on Medium]]></description>
        <link>https://medium.com/@banulakumarage?source=rss-312c941d4d02------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*KGE2cvvgbWxUkmMMQNCmzw.jpeg</url>
            <title>Stories by Banula Kumarage on Medium</title>
            <link>https://medium.com/@banulakumarage?source=rss-312c941d4d02------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 07:51:13 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@banulakumarage/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building a Real-Time ETL Pipeline with Apache Beam and Kafka — Part 2: Processing and Persisting]]></title>
            <link>https://medium.com/@banulakumarage/building-a-real-time-etl-pipeline-with-apache-beam-and-kafka-part-2-processing-and-persisting-fa41d03756e3?source=rss-312c941d4d02------2</link>
            <guid isPermaLink="false">https://medium.com/p/fa41d03756e3</guid>
            <dc:creator><![CDATA[Banula Kumarage]]></dc:creator>
            <pubDate>Sun, 25 May 2025 19:16:53 GMT</pubDate>
            <atom:updated>2025-05-25T19:16:53.681Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building a Real-Time ETL Pipeline with Apache Beam and Kafka — Part 2: Processing and Persisting</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/1*WPUmvIcmV-Gsg-fgGUbvOQ.png" /></figure><p>In the first part of this blog, we have consumed the IoT events from a Kafka topic using Apache Beam. If you haven’t read it yet, you can check it out <a href="https://medium.com/@banulakumarage/building-a-real-time-etl-pipeline-with-apache-beam-and-kafka-part-1-consuming-events-74627f5465b4">here</a>. In this part, we will explore how we can process this data and persist in a database.</p><h3>Process and Filter data</h3><p>Assume we receive below 10 events.</p><pre>{&quot;deviceId&quot;:&quot;deviceA&quot;,&quot;temperature&quot;:85,&quot;unit&quot;:&quot;C&quot;}<br>{&quot;deviceId&quot;:&quot;deviceA&quot;,&quot;temperature&quot;:70.5,&quot;unit&quot;:&quot;C&quot;}<br>{&quot;deviceId&quot;:&quot;deviceA&quot;,&quot;temperature&quot;:82.3,&quot;unit&quot;:&quot;C&quot;}<br>{&quot;deviceId&quot;:&quot;deviceB&quot;,&quot;temperature&quot;:90.1,&quot;unit&quot;:&quot;C&quot;}<br>{&quot;deviceId&quot;:&quot;deviceC&quot;,&quot;temperature&quot;:80.4,&quot;unit&quot;:&quot;C&quot;}<br>{&quot;deviceId&quot;:&quot;deviceC&quot;,&quot;temperature&quot;:82.0,&quot;unit&quot;:&quot;C&quot;}<br>{&quot;deviceId&quot;:&quot;deviceD&quot;,&quot;temperature&quot;:75,&quot;unit&quot;:&quot;C&quot;}<br>{&quot;deviceId&quot;:&quot;deviceD&quot;,&quot;temperature&quot;:88.7,&quot;unit&quot;:&quot;C&quot;}<br>{&quot;deviceId&quot;:&quot;deviceB&quot;,&quot;temperature&quot;:65.4,&quot;unit&quot;:&quot;C&quot;}<br>{&quot;deviceId&quot;:&quot;deviceC&quot;,&quot;temperature&quot;:26.9,&quot;unit&quot;:&quot;C&quot;}</pre><p>Assume we want to extract the events that have a temperature value higher than 80. Then, we want to count the number of such events for each device. Based on the records above, we should get the following output:</p><pre>deviceA: 2<br>deviceB: 1<br>deviceC: 2<br>deviceD: 1</pre><p>Before going for the processing part, let’s understand another important concept related to this.</p><ul><li>Assume there are vehicles parked in a car park. Since the number of vehicles remains unchanged, we can count them at any time without observing any variation. This type of data is referred to as <strong><em>bounded data</em></strong>.</li><li>But let’s consider a different scenario: we want to count the number of vehicles passing through a tunnel. Without any time constraint, the number of vehicles can be infinite. This type of data is called <strong><em>unbounded data</em></strong>. To count or group such unbounded data, we need to apply a windowing strategy. For example, we can count the number of vehicles passing through the tunnel between 8:00 AM and 9:00 AM.</li></ul><p>Since these streaming events are also unbounded data, we need to apply a windowing strategy to count the number of records. Let’s walk through the steps to see how we can process this data.</p><p>After consuming the events, now we have a <em>PCollection</em> with Long type key and <em>IotEvent</em> type value. For the next steps, we don’t need the key part of the Kafka events. Therefore we apply below transformation.</p><pre>.apply(Values.create())</pre><p>Now, to handle continuous streaming data, we apply fixed time windows of 5 seconds to group incoming events.</p><pre>.apply(Window.&lt;IotEvent&gt;into(FixedWindows.of(Duration.standardSeconds(5))))</pre><p>Now we can filter the events with below transformation.</p><pre>.apply(&quot;Filter events&quot;, ParDo.of(new EventsFilter()))</pre><p>Implementation of EventsFilter is as follows.</p><pre>public class EventsFilter extends DoFn&lt;IotEvent, String&gt; {<br><br>    @ProcessElement<br>    public void processElement(ProcessContext c) {<br>        IotEvent event = c.element();<br>        if (event.getTemperature() &gt; 80) {<br>            c.output(event.getDeviceId());<br>        }<br>    }<br>}</pre><p>Then we can apply below transformation to count the number of events per device ID in each window.</p><pre>.apply(Count.perElement())</pre><h3>Persist results</h3><p>Now, we have a key-value collection where the key is the device ID and the value is the number of records. We can persist this data in a PostgreSQL database using the <em>JdbcIO </em>beam connector.</p><pre>.apply(&quot;Persist events&quot;, JdbcIO.&lt;KV&lt;String, Long&gt;&gt;write()<br>       .withDataSourceConfiguration(JdbcIO.DataSourceConfiguration.create(<br>                               options.getDbDriverClass(),<br>                               options.getJdbcUrl()<br>                       )<br>                       .withUsername(options.getDbUserName())<br>                       .withPassword(options.getDbPassword())<br>       ).withStatement(&quot;INSERT INTO event(device_id, events_count)&quot; +<br>               &quot;VALUES (?, ?)&quot; +<br>               &quot;ON CONFLICT (device_id)&quot; +<br>               &quot;DO UPDATE SET events_count = event.events_count + EXCLUDED.events_count&quot;)<br>       .withPreparedStatementSetter(((element, preparedStatement) -&gt; {<br>           preparedStatement.setString(1, element.getKey());<br>           preparedStatement.setLong(2, element.getValue());<br>       })));</pre><p>This statement uses PostgreSQL’s ON CONFLICT clause for upserting records, so the count increments if the device already exists.</p><h3>Conclusion</h3><p>After executing the implemented Apache Beam real-time ETL pipeline, the consumed IoT from Kafka will be persisted in the Postgres database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/895/1*dl_A5XC0A16VFN7pSKNtQw.png" /></figure><p>The complete code for this real-time ETL pipeline is available <a href="https://github.com/BanulaKumarage/realtime-etl-with-apache-beam">here</a>.</p><p>At first, this may seem a bit complex. However, once you get familiar with Apache Beam, you’ll be able to implement much more advanced and complex scenarios with ease. To deepen your understanding, I would suggest you to refer to the official <a href="https://beam.apache.org/documentation/">Apache Beam documentation</a>.</p><p><em>Happy Reading!!!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fa41d03756e3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Real-Time ETL Pipeline with Apache Beam and Kafka — Part 1: Consuming Events]]></title>
            <link>https://medium.com/@banulakumarage/building-a-real-time-etl-pipeline-with-apache-beam-and-kafka-part-1-consuming-events-74627f5465b4?source=rss-312c941d4d02------2</link>
            <guid isPermaLink="false">https://medium.com/p/74627f5465b4</guid>
            <dc:creator><![CDATA[Banula Kumarage]]></dc:creator>
            <pubDate>Sun, 25 May 2025 18:55:25 GMT</pubDate>
            <atom:updated>2025-05-25T19:16:14.231Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building a Real-Time ETL Pipeline with Apache Beam and Kafka — Part 1: Consuming Events</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/1*4Kca4fdHt56hMCJWcwKIDQ.png" /></figure><p>ETL(Extract, transform, and load) jobs are used to process and move data from one place to another. Processing this data in real-time enables timely insights and actions. To handle this, we need scalable, fault-tolerant, and portable data pipelines.</p><p>Apache Beam is a powerful unified programming model for both batch and streaming data processing. In this blog, I will show how to build a real-time ETL pipeline with Apache Beam. A simple example is implemented here, where IoT sensor data is consumed from a Kafka topic, processed, and then persisted in a database. The Apache beam pipeline is as follows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*zGjLdY8v6NrUqbyLjaTWlg.png" /></figure><h3>Project Setup</h3><p>This ETL streaming processing application is implemented as a Maven project. After creating an empty Maven project, we start by setting up the project dependencies, including Apache Beam, Lombok(for annotations), Kafka, and PostgreSQL JDBC driver in the pom.xmlfile.</p><pre>&lt;dependencies&gt;<br>   &lt;dependency&gt;<br>       &lt;groupId&gt;org.apache.beam&lt;/groupId&gt;<br>       &lt;artifactId&gt;beam-sdks-java-core&lt;/artifactId&gt;<br>       &lt;version&gt;2.19.0&lt;/version&gt;<br>   &lt;/dependency&gt;<br>   &lt;dependency&gt;<br>       &lt;groupId&gt;org.apache.beam&lt;/groupId&gt;<br>       &lt;artifactId&gt;beam-runners-direct-java&lt;/artifactId&gt;<br>       &lt;version&gt;2.19.0&lt;/version&gt;<br>   &lt;/dependency&gt;<br><br>   &lt;dependency&gt;<br>       &lt;groupId&gt;org.apache.beam&lt;/groupId&gt;<br>       &lt;artifactId&gt;beam-sdks-java-io-kafka&lt;/artifactId&gt;<br>       &lt;version&gt;2.17.0&lt;/version&gt;<br>   &lt;/dependency&gt;<br>   &lt;dependency&gt;<br>       &lt;groupId&gt;org.apache.kafka&lt;/groupId&gt;<br>       &lt;artifactId&gt;kafka-clients&lt;/artifactId&gt;<br>       &lt;version&gt;2.5.0&lt;/version&gt;<br>   &lt;/dependency&gt;<br><br>   &lt;dependency&gt;<br>       &lt;groupId&gt;org.apache.beam&lt;/groupId&gt;<br>       &lt;artifactId&gt;beam-sdks-java-io-jdbc&lt;/artifactId&gt;<br>       &lt;version&gt;2.4.0&lt;/version&gt;<br>   &lt;/dependency&gt;<br>   &lt;dependency&gt;<br>       &lt;groupId&gt;org.postgresql&lt;/groupId&gt;<br>       &lt;artifactId&gt;postgresql&lt;/artifactId&gt;<br>       &lt;version&gt;42.7.1&lt;/version&gt;<br>   &lt;/dependency&gt;<br><br>   &lt;dependency&gt;<br>       &lt;groupId&gt;org.projectlombok&lt;/groupId&gt;<br>       &lt;artifactId&gt;lombok&lt;/artifactId&gt;<br>       &lt;version&gt;1.18.38&lt;/version&gt;<br>       &lt;scope&gt;provided&lt;/scope&gt;<br>   &lt;/dependency&gt;<br><br>&lt;/dependencies&gt;<br><br><br>&lt;build&gt;<br>   &lt;plugins&gt;<br>       &lt;plugin&gt;<br>           &lt;groupId&gt;org.apache.maven.plugins&lt;/groupId&gt;<br>           &lt;artifactId&gt;maven-compiler-plugin&lt;/artifactId&gt;<br>           &lt;configuration&gt;<br>               &lt;annotationProcessorPaths&gt;<br>                   &lt;path&gt;<br>                       &lt;groupId&gt;org.projectlombok&lt;/groupId&gt;<br>                       &lt;artifactId&gt;lombok&lt;/artifactId&gt;<br>                       &lt;version&gt;1.18.38&lt;/version&gt;<br>                   &lt;/path&gt;<br>               &lt;/annotationProcessorPaths&gt;<br>           &lt;/configuration&gt;<br>       &lt;/plugin&gt;<br>   &lt;/plugins&gt;<br>&lt;/build&gt;</pre><h3>Setting pipeline options</h3><p>Before going for the ETL pipeline, we need to set a couple of configuration properties that are needed for Kafka and the PostgreSQL database.</p><p>For that, we create a properties file named application.properties in the src/main/resources/path.</p><pre>kafkaBootstrapServers=localhost:9092<br>kafkaTopic=&lt;topic-name&gt;<br>dbDriverClass=org.postgresql.Driver<br>jdbcUrl=jdbc:postgresql://localhost:5432/iot<br>dbUserName=&lt;username&gt;<br>dbPassword=&lt;password&gt;</pre><p>Then, we create an interface extending the PipelineOptionsinterface so that above configured properties can be used in the ETL pipeline.</p><pre>public interface CustomOptions extends PipelineOptions {<br>   void setKafkaBootstrapServers(String bootstrapServers);<br>   String getKafkaBootstrapServers();<br><br>   void setKafkaTopic(String topic);<br>   String getKafkaTopic();<br><br>   void setDbDriverClass(String dbDriverClass);<br>   String getDbDriverClass();<br><br>   void setJdbcUrl(String jdbcUrl);<br>   String getJdbcUrl();<br><br>   void setDbUserName(String dbUser);<br>   String getDbUserName();<br><br>   void setDbPassword(String dbPassword);<br>   String getDbPassword();<br>}</pre><p>To load the properties from the properties file to a CustomOptionsobject, we have created a loader as shown below.</p><pre>public class OptionsLoader {<br><br>    public static CustomOptions loadOptions() {<br>        Properties properties = new Properties();<br>        try (InputStream input = OptionsLoader.class.getClassLoader()<br>                .getResourceAsStream(&quot;application.properties&quot;)) {<br>            if (input == null) {<br>                throw new RuntimeException(&quot;application.properties not found&quot;);<br>            }<br>            properties.load(input);<br>        } catch (Exception e) {<br>            throw new RuntimeException(&quot;Failed to load application.properties&quot;, e);<br>        }<br><br>        String[] args = properties.entrySet().stream()<br>                .map(e -&gt; &quot;--&quot; + e.getKey() + &quot;=&quot; + e.getValue()).toArray(String[]::new);<br><br>        return PipelineOptionsFactory.fromArgs(args)<br>                .withValidation()<br>                .as(CustomOptions.class);<br>    }<br>}</pre><p>Then we can load the properties before starting the pipeline steps.</p><pre>CustomOptions options = OptionsLoader.loadOptions();</pre><h3>Consuming events from Kafka</h3><p>Now we are ready to consume the events from Kafka using the <em>KafkaIO</em> beam connector.</p><pre>Pipeline p = Pipeline.create();<br><br>p.apply(&quot;Consume IoT events&quot;, KafkaIO.&lt;Long, IotEvent&gt;read()<br>               .withBootstrapServers(options.getKafkaBootstrapServers())<br>               .withTopic(options.getKafkaTopic())<br>               .withKeyDeserializer(LongDeserializer.class)<br>               .withValueDeserializer(IotDeserializer.class)<br>               .withoutMetadata())</pre><p>As you can see, we have created two additional classes. Our Kafka event has a key of type Long and a value of type Json. After reading the JSON value, we deserialize it into an IotEventobject. The IotEvent class is defined as follows.</p><pre>@NoArgsConstructor<br>@AllArgsConstructor<br>@Data<br>public class IotEvent implements Serializable {<br>    private String deviceId;<br>    private Double temperature;<br>    private String unit;<br>}</pre><p>And also, we have to create a custom deserializer extending the Deserializer&lt;T&gt; interface.</p><pre>public class IotDeserializer implements Deserializer&lt;IotEvent&gt; {<br>    @Override<br>    public void configure(Map&lt;String, ?&gt; configs, boolean isKey) {<br>        Deserializer.super.configure(configs, isKey);<br>    }<br><br>    @Override<br>    public IotEvent deserialize(String s, byte[] bytes) {<br>        ObjectMapper mapper = new ObjectMapper();<br><br>        IotEvent iotEvent = null;<br>        try {<br>            iotEvent = mapper.readValue(bytes, IotEvent.class);<br>        } catch (IOException e) {<br>            System.out.println(e.getMessage());<br>        }<br><br>        return iotEvent;<br>    }<br><br>    @Override<br>    public IotEvent deserialize(String topic, Headers headers, byte[] data) {<br>        return Deserializer.super.deserialize(topic, headers, data);<br>    }<br><br>    @Override<br>    public void close() {<br>        Deserializer.super.close();<br>    }<br>}</pre><p>After this, we can consume a sample event as below.</p><pre>{&quot;deviceId&quot;:&quot;deviceA&quot;,&quot;temperature&quot;:85.2,&quot;unit&quot;:&quot;C&quot;}</pre><p>After consuming the events, we have a <em>PCollection</em> with Long type key and <em>IotEvent</em> type value.</p><p>Now, we can process this data and persist it. In the next part of this series, I’ll walk you through the processing and persisting part of the pipeline. We’ll also explore how to apply windowing and filtering and, finally, how to write the results into a PostgreSQL database.</p><p><em>Let’s meet again in Part 2 to continue building this real-time ETL pipeline!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=74627f5465b4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Magic Behind @Transactional]]></title>
            <link>https://medium.com/@banulakumarage/the-magic-behind-transactional-997e6e809e4c?source=rss-312c941d4d02------2</link>
            <guid isPermaLink="false">https://medium.com/p/997e6e809e4c</guid>
            <dc:creator><![CDATA[Banula Kumarage]]></dc:creator>
            <pubDate>Fri, 25 Apr 2025 20:34:16 GMT</pubDate>
            <atom:updated>2025-04-25T20:43:33.899Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IvgEZAZslzr7QAhKWcGkHw.png" /></figure><p>There are various ways of managing transactions in different frameworks and different languages. When it comes to the Spring framework, the <em>@Transactional</em> annotation can be used to manage transactions. Here, we don’t want to manually commit or roll back when using the <em>@Transactional</em> annotation. It takes care of handling the complete transaction flow. But it’s important to understand how<em> @Transactional</em> works internally and performs this magic of managing transactions.</p><p>So let’s dig into this step by step.</p><p>Imagine a scenario where a bank transfer happens from one account to another.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/288/1*rOs0GR4r4KGu4JnSjLitmw.png" /></figure><p>To prevent any data loss in case of any failure, we have to implement this as a single unit of execution. We can use the <em>@Transactional</em> annotation as below to implement this function.</p><pre>@Transactional<br>public void transfer(Account senderAccount, Account receiverAccount, double amount) {<br><br>   senderAccount.setBalance(senderAccount.getBalance() - amount);<br>   accountRepository.save(senderAccount);<br>  <br>   receiverAccount.setBalance(receiverAccount.getBalance() + amount);<br>   accountRepository.save(receiverAccount);<br>}</pre><h4>Internal Flow</h4><p>In Spring, we can use Aspect Oriented Programming(AOP) when we want to implement any logic before or after executing a specific function. Internally, <em>@Transactional</em> use this AOP with Around Advice. With that, it performs some logic before and after the method invocation.</p><p>With the <em>@Transactional</em> annotation, the following flow get executed when the above transfer method is called.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/162/1*ZYjYl2FK7CBC4yO4Wog54g.png" /></figure><h4>Proxy Class</h4><p>When the <em>transfer</em> method is called from the <em>AccountService</em> bean, a proxy class is created. Since the <em>AccountService</em> class is a concrete class and not a child class, a CGLIB proxy class is created. Refer <a href="https://docs.spring.io/spring-framework/reference/core/aop/introduction-proxies.html">this</a> to understand further about AOP Proxies.</p><p>This Proxy class is created with extending the <em>AccountService</em> class and overriding the transfer method. After that, the invoke function in <em>TransactionInterceptor</em> is executed by passing the metadata of the target function (transfer function). A pseudo code for the proxy class can be illustrated as below (This is not the actual implementation).</p><pre>public class AccountService$$SpringCGLIB$$0 extends AccountService{<br><br>    private TransactionInterceptor transactionInterceptor;<br><br>    private AccountService targetClass;<br><br>    public AccountService$$SpringCGLIB$$0(AccountRepository accountRepository,<br>                               TransactionInterceptor transactionInterceptor,<br>                               AccountService targetClass) {<br>        super(accountRepository);<br>        this.transactionInterceptor = transactionInterceptor;<br>        this.targetClass=targetClass;<br>    }<br><br>    @Override<br>    public void transfer(Account senderAccount, Account receiverAccount, double amount) {<br>        try {<br>            Method method = AccountService.class.getMethod(&quot;transfer&quot;, Account.class, Account.class, double.class);<br><br>            MethodInvocation invocation = new MethodInvocation() {<br>                @Override<br>                public Method getMethod() {<br>                    return method;<br>                }<br><br>                @Override<br>                public Object[] getArguments() {<br>                    return new Object[]{senderAccount, receiverAccount, amount};<br>                }<br><br>                @Override<br>                public Object proceed() throws Throwable {<br>                    return targetClass.transfer(senderAccount, receiverAccount, amount);<br>                }<br><br>                @Override<br>                public Object getThis() {<br>                    return targetClass;<br>                }<br><br>                @Override<br>                public AccessibleObject getStaticPart() {<br>                    return method;<br>                }<br><br><br>            };<br><br>            transactionInterceptor.invoke(invocation);<br><br>        } catch (Throwable ex) {<br>            throw new RuntimeException(&quot;Transaction failed&quot;, ex);<br>        }<br>    }<br>}</pre><p>Inside the <em>transactionInterceptor</em>, the invocation is used to fetch the metadata of the target method.</p><h4>Inside the TransactionInterceptor</h4><p>Around Advice is applied to the target method inside this <em>TransactionInterceptor</em>. Before the target method invocation, transaction-related attributes are fetched using the method invocation that has been passed by the proxy. Then the target method gets executed. After that transaction is committed if there is no exception raised during the invocation. If any exception is raised, the transaction is rolled back. The main steps that happen inside the transactionInterceptor can be illustrated as below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/384/1*rpzqRIYNmIhM8OR2iPQD8Q.png" /></figure><blockquote><strong>Debug Tip:</strong></blockquote><blockquote>&gt; To understand this clearly, you can put a breakpoint in the <strong>TransactionInterceptor.invoke</strong> method, and you will be able to trace this full flow of transaction management.</blockquote><p>So the next time you annotate a method with <em>@Transactional</em>, you’ll know it’s not magic — it’s just smart design, proxies, and a little bit of Spring wizardry under the hood.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=997e6e809e4c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ways of Deploying in Kubernetes]]></title>
            <link>https://medium.com/@banulakumarage/ways-of-deploying-in-kubernetes-14763bc0f090?source=rss-312c941d4d02------2</link>
            <guid isPermaLink="false">https://medium.com/p/14763bc0f090</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[deployment]]></category>
            <dc:creator><![CDATA[Banula Kumarage]]></dc:creator>
            <pubDate>Thu, 28 Dec 2023 13:24:11 GMT</pubDate>
            <atom:updated>2023-12-28T13:24:11.995Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*saY3L3MQoZjp3DSZY8OMcw.png" /></figure><p>When deploying an application in a Kubernetes cluster, it can be changed over time. Deployment strategy defines the way of creating, upgrading and downgrading the versions of the deployed application in the cluster. Multiple deployment strategies can be implemented in a Kubernetes environment using its core components. Among them, we can discuss 4 major strategies and the way of implementation in a Kubernetes Environment.</p><h3>Rolling Update</h3><p>This is the default deployment strategy in Kubernetes. Here, when the application version is changed, the pods will be replaced with the newer version one by one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y8Wp-6dOwLVyOVWrisHQSQ.gif" /><figcaption>Rolling Update Process</figcaption></figure><p>This update can be a change of any of the properties in the Kubernetes Deployment such as image, number of replicas, labels, etc. This can be observed by the events in the deployments. To simulate that, first, we create a deployment with 3 replicas and <em>nginx:1.7.1</em> image.</p><pre>$ kubectl create deployment nginx --image=nginx:1.7.1 --replicas=3<br>deployment.apps/nginx created</pre><p>Then we change the image of the deployment to <em>nginx:1.9.1</em></p><pre>$ kubectl set image deployment.apps/nginx nginx=nginx:1.9.1<br>deployment.apps/nginx image updated</pre><p>During the rollout process, we can check the status of the update as below.</p><pre>$ kubectl rollout status deployment.apps/nginx<br>Waiting for deployment &quot;nginx&quot; rollout to finish: 0 of 3 updated replicas are available...<br>Waiting for deployment &quot;nginx&quot; rollout to finish: 1 of 3 updated replicas are available...<br>Waiting for deployment &quot;nginx&quot; rollout to finish: 2 of 3 updated replicas are available...<br>deployment &quot;nginx&quot; successfully rolled out</pre><p>Now when we check the events of the deployment using kubectl describe deployment nginx command, a pod is replaced one after the other.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6z6Gv8AZpRV6bjTg" /></figure><p>In the rollback process, Kubernetes creates a new Replica Set with the updated pods under the same deployment while taking down the pods in the earlier Replica Set one after the other.</p><p>If we want to go back to the previous version of the deployment, we can use the below command.</p><pre>$ kubectl rollout undo deployment/nginx<br>deployment.apps/nginx rolled back</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YLd1RYHQR8yJkkyt" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EYn8ZigOQTRykJsc" /><figcaption>Before(Left) and After(Right) states of the Replica Sets</figcaption></figure><h3>Recreate</h3><p>In this upgrade strategy, running instances of the older version are destroyed and then the new instances are deployed. During the process between destroying the older version and deploying the newer version, there is downtime which can be considered a disadvantage in this method.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CxSNuo6-pi7DP1GIRqjYLA.gif" /><figcaption>Recreating Process</figcaption></figure><p>To simulate that, let’s create the same deployment by defining the .spec.strategy.type as <em>Recreate</em> in the <em>yaml</em>. We can create the <em>yaml</em> file using the same command with some minor modifications as below.</p><pre>$ kubectl create deployment nginx --image=nginx:1.7.1 --replicas=3 -n blog --dry-run=client -o yaml &gt; deployment-definition.yaml</pre><p>Then we can update the deployment strategy as below. Observe, that the .spec.strategy.type property is set to <em>Recreate</em>.</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  labels:<br>    app: nginx<br>  name: nginx<br>spec:<br>  replicas: 3<br>  selector:<br>    matchLabels:<br>      app: nginx<br>  strategy:<br>    type: Recreate<br>  template:<br>    metadata:<br>      labels:<br>        app: nginx<br>    spec:<br>      containers:<br>      - image: nginx:1.7.1<br>        name: nginx</pre><p>After updating, we can create the deployment using kubectl create -f deployment-definition.yaml the command. Now we can change the image as we have done earlier and observe the deployment events.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gQVTT6vig4iQRGZJ" /></figure><p>We can observe that the whole Replica Set was scaled down to 0 and then the Replica Set with a newer version of pods was scaled up to 3.</p><p>Above mentioned deployment strategies were out-of-the-box mechanisms that were provided by Kubernetes. But using the concepts in Kubernetes we can use some other strategies as well such as the Blue/Green strategy and Canary Deployment strategy.</p><h3>Blue/Green Strategy</h3><p>In this strategy, the newer version of pods will be deployed alongside the older version. But the traffic is still routed to the older version. The older version is labelled as Blue and the newer version is labelled as Green. Tests are executed on the Green version. When all the tests are passed, traffic is directly switched to the Green version.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h-A4f3F7VhFi4LXNyXMWzQ.gif" /><figcaption>Blue/Green Upgrade Process</figcaption></figure><p>Service Meshes like <em>Istio</em> can be used to implement this in a better way. But using the core components, we can still simulate this in the Kubernetes environment. We have a deployment with the label with <em>version:v1</em>. The pods in the deployment will be exposed using the <em>v1</em> label by using it in the .spec.selector section. Then we deploy <em>v2</em> version with the label of <em>version:v2</em>. Once all the tests are passed in the <em>v2</em> version, we change the selector of the service to <em>v2</em> so that all the traffic is switched to the <em>v2</em> version of deployment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*96FkoJYZgCXvuf36WVj6pA.gif" /></figure><h3>Canary Update</h3><p>In this strategy, we deploy the new version and route only a small percentage of traffic to the newer version. If all the tests are passed and work as expected, all the instances of the older version are upgraded to the new version and canary deployment will be destroyed. For that, we can use a default strategy like RollingUpdate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/1*j0iz1KXbtxfoEDgwsI0ryQ.gif" /><figcaption>Process in Canary Update</figcaption></figure><p>In the implementation, first, we create a deployment with label <em>version:v1</em> and expose the pods with a selector as <em>version:v1</em>. When a pod of a newer version is deployed, we give a common label <em>app:front-end</em> to both of the deployments. And then the selector of the service is changed to <em>app:front-end</em> so that traffic will be routed to both of the deployments.</p><p>The percentage of the traffic passing to the canary deployment will be controlled via the number of replicas in the canary deployment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/943/0*ZKyDw5n_YA0MG3Jv" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/943/0*Y19ujge9qP0IACOt" /></figure><p>In this manner of implementation, the percentage of traffic is always proportional to the number of pods. But this can be done in a better way with service meshes like Istio</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=14763bc0f090" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Helm: Kubernetes Package Manager]]></title>
            <link>https://medium.com/@banulakumarage/helm-kubernetes-package-manager-8e475594780f?source=rss-312c941d4d02------2</link>
            <guid isPermaLink="false">https://medium.com/p/8e475594780f</guid>
            <dc:creator><![CDATA[Banula Kumarage]]></dc:creator>
            <pubDate>Sun, 26 Feb 2023 20:20:49 GMT</pubDate>
            <atom:updated>2023-02-26T20:20:49.224Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*vYjNPycxLPi6nv7fDPoBwQ.png" /></figure><p>If you have ever used a package manager like Yum, Chocolatey or APT, you can understand how it’s easy to install packages using a package manager without spending time on going through the GitHub repos to find a specific package. Helm is exactly a package manager which is specifically for a Kubernetes cluster. There are a larger number of pre-configured software packages that can be found in package repositories like <a href="https://artifacthub.io/"><em>Artifact Hub</em></a><em>.</em></p><p>As an example, if we want to install MySQL on our Kubernetes Cluster without using Helm, we have to do a lot of things like getting the official Docker image from <a href="https://hub.docker.com/">Docker Hub</a>, writing the pod configuration YAMLs, writing the service configurations etc. But with Helm we can easily install MySQL into our cluster with a couple of simple commands. Not only that, Helm can be used to install a collection of software as a single package. The whole collection of packages can be managed as a single software without worrying about the underlying each deployment. <a href="https://artifacthub.io/packages/helm/prometheus-community/kube-prometheus-stack">Kube-prometheus-stack</a> is such an example which includes all the monitoring needs in a single package.</p><h3>Install Helm</h3><p>To work with helm first we need to install Helm. You can download the specific binary file which is compatible for your operating system using the <a href="https://github.com/helm/helm/releases">Official Releases</a> page. Then you just need to extract the downloaded folder and add the path to helm.exe in the path environment variable. To verify the installation, just run the following command and check you will get a similar output.</p><pre>$ helm version<br>version.BuildInfo{Version:&quot;v3.11.0-rc.2&quot;, GitCommit:&quot;472c5736ab01133de504a826bd9ee12cbe4e7904&quot;, GitTreeState:&quot;clean&quot;, GoVersion:&quot;go1.18.9&quot;}</pre><h3>Install Helm Charts</h3><p>Let’s take MySQL as an example. To install the MySQL package in the cluster, the package has to be found in a package repository. Here I’m using <a href="https://artifacthub.io/"><em>Artifact Hub</em></a>. I’m going to install <a href="https://artifacthub.io/packages/helm/bitnami/mysql">MySQL by Bitnami</a> into the cluster. For that we can use the following commands.</p><pre>$ helm repo add bitnami-repo https://charts.bitnami.com/bitnami<br>$ helm install mysql-release bitnami-repo/mysql</pre><p>In the first command we are making a repository in our local machine to store the <a href="https://charts.bitnami.com/bitnami.">https://charts.bitnami.com/bitnami.</a> bitnami-repo is a custom repo name which you can change according to your preference. Again in the second command, mysql-release is a custom label which is used to refer to this installation at a later date. The installation is placed inside the bitnami-repo. You can list the repos you have created using the following command.</p><pre>$ helm repo list</pre><p>In Helm, chart is a term used to describe a single package installation. You can list the installed helm charts using the following command.</p><pre>$ helm list</pre><p>After successfully installing MySQL, you can see all the relevant deployment components by using the following kubectl command.</p><pre>$ kubectl get all<br>NAME                  READY   STATUS    RESTARTS   AGE<br>pod/mysql-release-0   0/1     Running   0          30s<br><br>NAME                             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE<br>service/mysql-release-headless   ClusterIP   None            &lt;none&gt;        3306/TCP   30s<br>service/mysql-release            ClusterIP   10.152.183.66   &lt;none&gt;        3306/TCP   30s<br><br>NAME                             READY   AGE<br>statefulset.apps/mysql-release   0/1     30s</pre><p>When we want to uninstall this MySQL package, we can use the following helm command and uninstall all the relevant above-displayed components without thinking of removing each component.</p><pre>$ helm uninstall mysql-release</pre><p>You can also remove the repo using following command.</p><pre>$ helm repo remove bitnami-repo</pre><h3>Overriding the default configuration with custom values</h3><p>With Helm, we can change the configurations of a package with our custom values ( ex. : username/ password, port that runs the service, naming convention of pods, etc.) For that we have to get the key value pairs regarding the configuration. As an example we can get the configuration of the MySQL package which was installed earlier to the YAML file named <em>values.yaml</em> using the following command.</p><pre>$ helm show values bitnami-repo/mysql &gt; values.yaml</pre><p>The configuration of the MySQL helm chart is saved to the <em>values.yaml</em> file with all the key value pairs. As in the below image, in <em>values.yaml, </em>the <em>rootPassword </em>of MySQL is mentioned. Let’s think, we want to change it to <em>“root”</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3McFIoAU3UKRiTPP" /></figure><p>We can give this custom value and upgrade the current mysql installation using the below command.</p><pre>$ helm upgrade mysql-release bitnami-repo/mysql --set auth.rootPassword=root</pre><p>We have to be careful when accessing keys in the YAML, since it is hierarchical.</p><p>If we want to override multiple values, we can create a YAML file and pass it to the helm installation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QtcBVSfo-3FxgYw7Me-jfg.png" /></figure><p>Let’s think we want to change the name of the database and the rootPassword. Then we can create a custom YAML file as below which is named as <em>custom_values.yaml </em>in this example, and pass it. When adding those customizations, you can refer to the official documentation of the relevant helm chart.</p><pre>auth:<br>  rootPassword: &quot;&quot;<br>  database: &quot;custom_db&quot;</pre><p>We can pass this YAML using the following command.</p><pre>$ helm upgrade mysql-release bitnami-repo/mysql --values=custom_values.yaml</pre><p>Without using the remote repo all the time, we can pull the remote repo to the local machine to install packages. To download and extract the folder we can use the following command.</p><pre>$ helm pull bitnami-repo/mysql --untar=true</pre><p>After pulling the package to the local machine, we can use the package using this folder using the following command.</p><pre>$ helm install mysql-release ./mysql/</pre><p>As we did earlier, we can override the values with custom values and upgrade the installation using this locally saved folder as well. The command is as follows.</p><pre>$ helm upgrade mysql-release --values=custom_values.yaml ./mysql</pre><p>./mysql is the relative path to the pulled folder.</p><p>We can generate a YAML using those custom values and directly apply to the cluster as well using kubectl. To generate the values and save them in the <em>mysql.yaml </em>file<em>, </em>you can use the following command.</p><pre>$ helm template ./mysql/ --values=custom_values.yaml &gt; mysql.yaml</pre><p>Now we can apply this YAML to the cluster.</p><pre>$ kubectl apply -f mysql.yaml</pre><p>The disadvantage of this method is that we have lost the control over this installation. If we want to remove the components, we have to do it one by one manually.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8e475594780f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>