<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Arunan Senthil on Medium]]></title>
        <description><![CDATA[Stories by Arunan Senthil on Medium]]></description>
        <link>https://medium.com/@senthilarunan95?source=rss-122bac81f141------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*LJ_vM4Q_XfSijASA</url>
            <title>Stories by Arunan Senthil on Medium</title>
            <link>https://medium.com/@senthilarunan95?source=rss-122bac81f141------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 21:59:46 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@senthilarunan95/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Kafka Connect Series: Quick Introduction]]></title>
            <link>https://medium.com/@senthilarunan95/kafka-connect-series-quick-introduction-eb90f6867a63?source=rss-122bac81f141------2</link>
            <guid isPermaLink="false">https://medium.com/p/eb90f6867a63</guid>
            <category><![CDATA[converter]]></category>
            <category><![CDATA[smt]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[kafka-connector]]></category>
            <category><![CDATA[kafka-connect]]></category>
            <dc:creator><![CDATA[Arunan Senthil]]></dc:creator>
            <pubDate>Thu, 18 Jul 2024 17:14:12 GMT</pubDate>
            <atom:updated>2024-07-18T17:22:32.914Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RMEn68LcS4XW3JsMBL1u4g.png" /><figcaption>Kafka Connect</figcaption></figure><p>Data plays a key role in almost every sector in the world. Vast amounts of data are generated over time. To gain valuable insights from that data, we need to process it and transform it into information. We cannot do this directly from the sources where the data originates. Instead, we need to transfer the data to a convenient environment to transform or store it. For that transition, we can use Apache Kafka. Apache Kafka is a distributed streaming platform that follows a publisher-subscriber architecture.</p><p>Kafka has impressive qualities, making it a prominent example of a distributed streaming platform:</p><ol><li>Real-time messaging</li><li>Scalability</li><li>Fault tolerance</li><li>Real-time processing</li><li>Durability</li></ol><p>Log aggregation, real-time pipelines, and event sourcing are some of the major use cases of Kafka.</p><p>following article is more helpful to know about kafka before dive into kafka connect</p><p><a href="https://medium.com/@rajithaf/theory-behind-kafka-30d21b64a9de">Theory Behind Kafka. Today, we are going to discuss the… | by Rajitha Fernando | Medium</a></p><p>Kafka Connect is an important component of the Apache Kafka ecosystem, used to integrate Kafka with external data systems. In simple terms, Kafka Connect brings data into Kafka from external systems and takes data out of Kafka to external systems.</p><p>Kafka Connect has three main concepts: connectors, tasks, and workers. A connector is a piece of code (primarily Java JARs) that defines the logic for data integration. Connectors assign tasks (units of work to be done) based on user configurations. A worker is a Java process that executes these tasks.</p><p>Kafka Connect can be run in two modes: standalone and distributed. In standalone mode, a single Kafka Connect worker handles all tasks. In distributed mode, multiple workers collaborate to share the workload. Connectors assign tasks to Kafka Connect workers, which manage the data transfer between Kafka and external systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sAGUv0iCvcjZtQDa" /></figure><p>A connector creates the connection between a source or sink system and Kafka. The connector handles tasks such as establishing connections and performing authorization.</p><p>Single Message Transformations (SMTs) allow for the transformation of individual messages as they pass through Kafka Connect. These transformations can include actions like masking sensitive data, adding new columns, and removing existing columns.</p><p>A converter is responsible for serialization in source connectors and deserialization in sink connectors. It converts messages into a specific format, such as AVRO or JSON.</p><p>I hope this guide was helpful. Please let me know in case of any questions or comments. Have a nice day</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eb90f6867a63" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>