<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Isha Gupta on Medium]]></title>
        <description><![CDATA[Stories by Isha Gupta on Medium]]></description>
        <link>https://medium.com/@ishagupta_75770?source=rss-460ce8711be5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*-uy8qpSpJSIXF8uDAdraaQ.png</url>
            <title>Stories by Isha Gupta on Medium</title>
            <link>https://medium.com/@ishagupta_75770?source=rss-460ce8711be5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 18:58:52 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ishagupta_75770/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[From Central Data Teams to Data Mesh: Lessons from the Microservices Playbook]]></title>
            <link>https://medium.com/@ishagupta_75770/from-central-data-teams-to-data-mesh-lessons-from-the-microservices-playbook-06a84a178dd0?source=rss-460ce8711be5------2</link>
            <guid isPermaLink="false">https://medium.com/p/06a84a178dd0</guid>
            <dc:creator><![CDATA[Isha Gupta]]></dc:creator>
            <pubDate>Mon, 26 May 2025 07:06:49 GMT</pubDate>
            <atom:updated>2025-05-26T23:24:04.842Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A reflection on evolving data architectures — what we can learn from the microservices movement.</em></p><p>As organisations grow, their data needs become more complex. Centralised data teams, which initially serve well, can become bottlenecks, struggling to meet diverse demands from various domains. This evolution mirrors what we saw in software engineering — from monoliths to microservices — now playing out in the data world as a shift toward <strong>Data Mesh</strong>.</p><h3>The Challenges of Centralised Data Teams</h3><p>Centralised data teams often face several challenges as organisations scale:</p><ul><li><strong>Backlogs of data requests</strong>: Handling diverse requirements from multiple domains can lead to delays.</li><li><strong>Limited domain context</strong>: A central team may lack the specific knowledge needed for domain-specific data tasks.</li><li><strong>Conflicting priorities</strong>: Balancing various stakeholder needs can be challenging, leading to slow turnaround on changes and frustrated domain teams that lack visibility.</li><li><strong>Operational overload</strong>: Managing on-call duties and incident responses can lead to burnout.</li><li><strong>Scalability pressures</strong>: As a company grows, the need to scale quickly becomes critical. But when a central data team is overwhelmed, progress slows. Teams want to fix issues and pay down tech debt — but simply don’t have the bandwidth.</li><li><strong>Migration complexity</strong>: Ongoing migrations — like moving between the data centres or cloud, adopting new warehouses, or refactoring pipelines — further stretch already limited resources.</li></ul><h3>Understanding Data Mesh</h3><p><strong>Data Mesh</strong> is a decentralised approach to data architecture. This approach aims to align data ownership with domain expertise, promoting agility and scalability. Instead of a single team managing all data assets, domain teams own and maintain their datasets as <strong>data products</strong>. It’s not just a technical shift — it’s a cultural and organisational change, requiring alignment, enablement, and buy-in from leadership.</p><p>Ideally, each team is responsible for:</p><ul><li>Producing clean, reliable data</li><li>Making it discoverable and accessible</li><li>Maintaining documentation and contracts</li></ul><h3>Benefits:</h3><ul><li>Data is closer to the people who understand it</li><li>Enables self-service for downstream teams</li><li>Reduces the load on centralised data teams</li></ul><h3>Challenges:</h3><ul><li>Needs good observability and governance tools</li><li>Requires alignment on standards across domains</li><li>Cultural shift: not every team is ready for this added responsibility to own data</li></ul><h3>Drawing Parallels: Monoliths vs. Microservices and Centralised Data Teams vs. Data Mesh</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1Cbev6590qncQCN1-d7vaw.png" /></figure><p>In Software architecture:</p><p><strong>Monolithic Systems:</strong></p><ul><li>Centralised control</li><li>Tightly coupled components</li><li>Single point of failure</li><li>Slower, complex deployments</li></ul><p><strong>Microservices:</strong></p><ul><li>Distributed ownership</li><li>Loosely coupled via APIs</li><li>Greater resilience through service isolation</li><li>Enables continuous delivery</li></ul><p>Now , in Data architecture:</p><p><strong>Centralised Data Teams:</strong></p><ul><li>A single team manages all data</li><li>Often becomes a bottleneck</li><li>Limited understanding of domain-specific context</li><li>Difficult to scale effectively as organisation grows</li></ul><p><strong>Data Mesh:</strong></p><ul><li>Data ownership lies with domain teams</li><li>Allows for parallel development and ownership</li><li>Deep domain expertise leads to better data quality and reliability</li><li>More scalable and adaptable to changing needs</li></ul><p>In essence, centralisation offers control but can be less adaptable, while distributed ownership provide flexibility and speed but require strong coordination and governance,</p><h3>Real-World Implementations</h3><p><a href="https://netflixtechblog.com/data-mesh-a-data-movement-and-processing-platform-netflix-1288bcab2873">Netflix</a> adopted a Data Mesh architecture for increased flexibility and scale in their data pipelines.</p><p><a href="https://www.uber.com/en-AU/blog/datamesh/">Uber</a> restructured their data platform with a domain-oriented model to decentralise ownership, improve governance, and self-serve infrastructure.</p><p>The holistic approach, as demonstrated in case studies, reveals that the benefits of a decentralised, domain-oriented data architecture are contingent upon widespread organisational buy-in and a commitment to consistent data governance practices.</p><p>Conversely, they also highlight the potential challenges of a Data Mesh without crucial non-technical elements that can result in data silos, inconsistencies, and increased complexity.</p><p>Successful Data Mesh adoption needs a cultural transformation alongside the implementation of global data standards and the underlying technical infrastructure. From a centralized control to domain-oriented ownership inherently involves organizational and cultural change, which is always a non-trivial challenge.</p><h3>Considerations for Transitioning</h3><p>Transitioning to a Data Mesh architecture involves both technical and cultural changes. I think some steps to carefully consider:</p><ol><li><strong>Assess Organisational Readiness</strong>: Do teams have the technical maturity and support to own their data?</li><li><strong>Invest in Infrastructure</strong>: Provide self-serve tools that reduce dependency on central teams.</li><li><strong>Establish Clear Governance</strong>: Set standards and policies to maintain data quality and security.</li><li><strong>Foster a Cultural Shift</strong>: Encourage product thinking and empower teams to treat data like a first-class citizen.</li></ol><p>It’s essential to recognise that Data Mesh is not a one-size-fits-all solution. The right approach depends on your organisation’s size, complexity, maturity, and long-term goals.</p><h3>Conclusion</h3><p>The evolution from centralised data teams to a Data Mesh architecture reflects a broader trend towards decentralisation, mirroring the microservices transformation in software engineering. While centralised models offer consistency and control, they often struggle under the weight of scale and demand. Data Mesh, on the other hand, promises flexibility and speed — but at the cost of added complexity and a need for strong cultural alignment.</p><p>This isn’t a debate of one being better than the other. It’s about understanding trade-offs and choosing what fits best — today and tomorrow. Whether you stick with a centralised model or start exploring Data Mesh, what matters most is that your data architecture is able to evolve in tandem with your organisation’s needs.</p><h3>Credits &amp; References</h3><p><strong>Referenced Case Studies &amp; Articles</strong></p><ul><li><a href="https://netflixtechblog.com/data-mesh-a-data-movement-and-processing-platform-netflix-1288bcab2873">Data Mesh — A Data Movement and Processing Platform @ Netflix</a></li><li><a href="https://www.uber.com/en-AU/blog/datamesh/">DataMesh: How Uber laid the foundations for the data lake cloud migration</a></li></ul><p><strong>Conceptual Influences</strong></p><ul><li><a href="https://martinfowler.com/articles/data-mesh-principles.html">Zhamak Dehghani</a>’s foundational work on Data Mesh Principles</li><li>Evolution of microservices from monolithic architectures, as documented by industry leaders at ThoughtWorks and <a href="https://martinfowler.com/">Martin Fowler</a>.</li></ul><p>Feel free to connect if you’re navigating a similar journey — we can learn a lot from shared stories.</p><p><em>This post was developed with the help of ChatGPT, which helped in refining structure, tone, and clarity.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=06a84a178dd0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to use Confluent HTTP Sink Connector]]></title>
            <link>https://medium.com/@ishagupta_75770/how-to-use-confluent-http-sink-connector-419565c88146?source=rss-460ce8711be5------2</link>
            <guid isPermaLink="false">https://medium.com/p/419565c88146</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[kafka-connect]]></category>
            <category><![CDATA[confluent-kafka]]></category>
            <category><![CDATA[event-streaming]]></category>
            <category><![CDATA[kafka-sink-connect]]></category>
            <dc:creator><![CDATA[Isha Gupta]]></dc:creator>
            <pubDate>Tue, 28 Apr 2020 08:21:03 GMT</pubDate>
            <atom:updated>2022-07-19T13:25:20.017Z</atom:updated>
            <content:encoded><![CDATA[<h3>Connector to pull data <em>from</em> KAFKA <em>to</em> a REST API.</h3><p>Confluent supports a <a href="https://www.confluent.io/hub/#HTTP">HTTP-Sink-Connector</a> to integrate Apache Kafka with an API via HTTP or HTTPS. This connector is available in a Sink configuration i.e. to pull data from Kafka to a downstream REST API .</p><p>All this information with code is also available in <a href="https://github.com/guptais/demo-http-sink-kafka-connector">github</a>.</p><p><a href="https://docs.confluent.io/current/connect/kafka-connect-http/connector_config.html#license-topic-configuration">Note: This connector needs a Confluent Enterprise license.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EEKnxpTdgpS-E4M4V7590A.png" /><figcaption>An example use case</figcaption></figure><h3>Problem statement:</h3><ul><li><strong>Consume</strong> a message from a topic and <strong>Transform</strong> the message fields for the REST API</li><li><strong>POST</strong> it to a REST API via HTTP or HTTPs.</li><li><strong>Capture</strong> the API response.</li><li><strong>Record</strong> any errors while posting the message.</li></ul><h3>Prerequisites:</h3><p>Confluent Platform is installed and services are running locally.</p><p>There are multiple ways you can run a local KAFKA cluster. Refer the Confluent Quickstart link below. For the sake of the demo, I am using Confluent Hub local.</p><h3>Install http connector using confluent hub client.</h3><pre>confluent-hub install confluentinc/kafka-connect-http:latest</pre><h3>Demo REST API service</h3><p>Start any REST API. <em>I used the one by Confluent </em><em>kafka-connect-http-demo</em></p><pre>git clone https://github.com/confluentinc/kafka-connect-http-demo.git<br>cd kafka-connect-http-demo<br>mvn spring-boot:run -Dspring.profiles.active=simple-auth</pre><h3>Start the local Kafka cluster</h3><pre>confluent local start</pre><p>Create the connector configuration file ‘simpleHttpSink.json’.</p><pre>{<br> &quot;name&quot;: &quot;SimpleHttpSink&quot;,  <br> &quot;config&quot;: <br> {<br>   &quot;topics&quot;: &quot;http-messages&quot;,    <br>   &quot;tasks.max&quot;: &quot;1&quot;,    <br>   &quot;connector.class&quot;: &quot;io.confluent.connect.http.HttpSinkConnector&quot;,<br>   &quot;http.api.url&quot;: &quot;http://localhost:8080/api/messages&quot;,<strong><br>   </strong>&quot;value.converter&quot;:<br>       &quot;org.apache.kafka.connect.storage.StringConverter&quot;,    <br>   &quot;confluent.topic.bootstrap.servers&quot;: &quot;broker:9092&quot;,<br>   &quot;confluent.topic.replication.factor&quot;: &quot;1&quot;,<br>   &quot;reporter.bootstrap.servers&quot;: &quot;broker:9092&quot;,<br>   &quot;reporter.result.topic.name&quot;: &quot;success-responses&quot;,<br>   &quot;reporter.result.topic.replication.factor&quot;: &quot;1&quot;,<strong><br>   </strong>&quot;reporter.error.topic.name&quot;: &quot;error-responses&quot;,<br>   &quot;reporter.error.topic.replication.factor&quot;: &quot;1&quot;<br> }<br>}</pre><p><em>Reporter</em> part of the configuration helps in capturing the http response and any errors occurred while posting the messages. Response messages are like key value pairs with associated schemas. Currently the connector supports <a href="https://docs.confluent.io/current/connect/kafka-connect-http/connector_config.html#formatter">JSON formatting</a>.</p><h3>Load the simple HTTP sink connector</h3><pre>confluent local load httpsink -- -d simpleHttpSink.json</pre><h3>Check status of connector</h3><pre>confluent local status SimpleHttpSink</pre><h4>Capturing the HTTP Response from the API</h4><p>Produce some string messages to the Source http-messages topic</p><pre>confluent local produce http-messages</pre><p>Consume the http response in the success-responses topic</p><p><em>Note: Open few consumers in another terminal window to monitor the other topics.</em></p><pre>confluent local consume &lt;topic-name&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5NE-C2kjh6WiZFpcUr3jxA.png" /></figure><h4>Capturing Failed POST requests:</h4><p>We can do error reporting to provide monitoring around failed requests by capturing http response codes as error messages.</p><p>Reproduce an error in the REST API, error is received in the error-responses topic. <em>(See in the bottom right corner in the screenshot)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3MhUJEC_oHuQ-t3lT4AheA.png" /></figure><p><em>Note: You might notice a lag in error reporting because of the retry attempts in POST. These are </em><a href="https://docs.confluent.io/current/connect/kafka-connect-http/connector_config.html#retries"><em>configurable</em></a><em>.</em></p><h4>Transformation of messages</h4><p><a href="https://www.confluent.io/blog/kafka-connect-single-message-transformation-tutorial-with-examples/">SMT</a>(Single Message Transform) can be used to do transformations like renaming, masking, etc. fields for the target systems. There are many pre-built transformations available. One can also write custom transformations.</p><p>For implementing any kind of stateful, complex transformations and operations, like aggregation, KSQL and Kafka Streams are the recommended libraries.</p><h3>Troubleshooting:</h3><ul><li>Below are some commands I used to debug some issues while developing this demo:</li></ul><pre>//Restart the connector<br>curl -X POST localhost:8083/connectors/SimpleHttpSink/restart | jq</pre><pre>//Unload a connector<br>confluent local unload SimpleHttpSink -- -d simpleHttpSink.json</pre><ul><li>You can also delete the connector and create it again. Refer the below links.</li><li>If you face an issue where your zookeeper is not starting from confluent local start. You can take a look at this <a href="https://github.com/confluentinc/confluent-cli/issues/33">github issue</a> with my fix.</li></ul><h3>References for further reading:</h3><ul><li><a href="https://docs.confluent.io/current/connect/kafka-connect-http/index.html#connect-http-connector">HTTP-Sink-Connector</a></li><li><a href="https://docs.confluent.io/current/quickstart/index.html">Confluent Quickstart</a></li><li><a href="https://docs.confluent.io/current/cli/command-reference/confluent-local/index.html#confluent-local">Confluent Local</a></li><li><a href="https://docs.confluent.io/3.2.0/connect/managing.html">Managing Kafka Connect</a></li><li><a href="https://www.confluent.io/blog/webify-event-streams-using-kafka-connect-http-sink/">Confluent blog</a></li></ul><p>With this post I am sharing the research, I did for a recent event streaming project with Confluent cloud. While doing so, I put together a demo which will work for anybody who wants to see how this connector works. All the information already exists in the Confluent documentation. In my experience of working with Apache Kafka and Confluent Cloud, there are often multiple ways to achieve an outcome. In this post I am just sharing one of the many ways to achieve an outcome.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=419565c88146" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>