<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vipin Thazhe Madam on Medium]]></title>
        <description><![CDATA[Stories by Vipin Thazhe Madam on Medium]]></description>
        <link>https://medium.com/@vipinvasudevan?source=rss-41f56dc06719------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*xcURHZIWA30GTfQk.</url>
            <title>Stories by Vipin Thazhe Madam on Medium</title>
            <link>https://medium.com/@vipinvasudevan?source=rss-41f56dc06719------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 18:57:31 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vipinvasudevan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Notifications system using Kafka and KSQL]]></title>
            <link>https://engg.glance.com/notifications-system-using-kafka-and-ksql-d05499220126?source=rss-41f56dc06719------2</link>
            <guid isPermaLink="false">https://medium.com/p/d05499220126</guid>
            <dc:creator><![CDATA[Vipin Thazhe Madam]]></dc:creator>
            <pubDate>Mon, 02 Dec 2024 08:30:16 GMT</pubDate>
            <atom:updated>2024-12-02T08:30:16.150Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*IVcBxCjWDsVS-3ZJI1Tjbw.png" /></figure><p><strong>Context</strong></p><p>Red dots — one of the greatest cognitive rewards in social media applications and elsewhere. Psychologically it is thought of as to yield a positive outcome always and it changed the trajectory of lot of applications and services. These dots indicate users that there is something new or updated is waiting for them. Bringing this new, updated information to users involved a bunch of machines to work together.</p><p>Nostra is a gaming platform, home to 200M+ gamers, 1000+ games and it comes as part of Glance, smart lock screen for Android devices. Nostra has <br>a great variety of free to play instant games, Android games, Gaming community etc.</p><p>Nostra needs to notify users when users’ post receives a comment, or someone reacted to a post, or a new game released or there are new levels released in the game. Nostra transcends lockscreen, app and multiple surfaces and users need to be informed about what is happening in their choice of Nostra product. This presented a need to build a notification system that transcends client applications and incorporate multiple use cases and expand and scale this new use cases in future.</p><p>We are building a pull-based notification system, which means the user has to request fetching notifications, unlike APNS, which pushes notifications to user devices.</p><p>To Begin with, let us start with a simple diagram that explains what we are trying to achieve.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dRyJ3LGnKhl_sddjAa90EA.png" /></figure><p><strong>Functional Requirements</strong></p><p>The key functional requirements for the system are as follows.</p><p>1.Relay notifications from Source(e.g. Gaming Community) to multiple surfaces like Nostra Game Centre, Nostra App, Web etc.</p><p>2.Aggregate similar notifications so that the user is not bombarded with messages of the same type. For example, Suppose you created a Post and people started liking it. Notifications should appear as UserA and 2 others liked your post and not UserA, UserB, and UserC liked your post.</p><p>3.Product requirements change and you should have the flexibility to relay combinations of different source notifications to different targets</p><p>4.At the target, classify the notification as Earlier (let’s say 72 hours), Read Notifications etc.We need to Store only the aggregate of the notification and not the entire notification history.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/392/1*Ps2YYNJCtWaokoCRBlqjrg.png" /></figure><p><strong>Chronosphere</strong></p><p>Chronosphere was designed to be the central hub for notifications for Nostra. The systems technology stack consists of Java Spring boot micro-service, Kafka for message streaming, KSQL for real-time data processing, Spanner sink connector and Spanner as a database.</p><p>We decided to go for a Kafka, KSQL-based solution for collection, transformation and aggregation of the incoming stream of events since we are expecting to receive them at a high scale and Kafka seems to be a good fit here. We needed a smart way to transform and aggregate this data on the fly and make it available for reads at a later point in time. Not only did we need simple aggregations but we also needed window-based aggregation as well. E.g. “Compute likes of a post in the last 72 hours”.</p><p>We decided against using the Redis-pub sub because it is designed for simple data processing and lacks the advanced stream processing capabilities that KSQL offers.</p><p>Also since both Apache Kafka and KSQL were shipped as part of the confluent cloud which we already used in our existing stacks we didn’t have to spend much time in the configuration and setup of these.</p><p>Our next choice was between KSQL vs Apache Flink. We decided to go with KSQL due to the team’s familiarity with SQL-based syntax and ease of use. Flink also would have suited our purpose.</p><p><strong>High Level Architecture of the System</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ef2OYNJvd_zdMb1u0FS-SQ.png" /></figure><p><strong>Data Ingestion Layer</strong></p><p>Notifications are generated at various sources. E.g</p><p>1)From the community, if someone liked a post.</p><p>2) From the game Publishing portal etc someone published a game or Instant Game Challenge.</p><p>Note that these sources are backed by individual micro-services that can transform data into the required notification payload format. These events are initially sent to the Chronosphere backend where it is enriched with details like user avatar, user display name etc. Then the messages are pushed into a Kafka Queue.</p><p><strong>Transformation and Aggregation Layer</strong></p><p>We have a large number of events coming at a high rate. We don’t need to store each and every event since the source already stores this information. What we need is an aggregated view.</p><p>We need to aggregate similar notifications so that the user is not bombarded with messages of the same type.</p><p>So for the notifications that need to be aggregated, we will generate a unique ID for all such events. For eg In the above case a notification Identifier like POST_ID+LIKE will be generated and we run a count query for this notification identifier.</p><p>Similarly, we will have unique IDs generated for comments, shares etc.</p><p>If we do not want the notification to be combined, eg “someone sent you a message”, we can simply choose to generate a unique identifier for that notification.</p><p>We have used a KSQL Persistent aggregate query which will compute the counts of each of the notifications. Persistent queries in KSQL are SQL statements that continuously process incoming data from Kafka topics and produce results for other Kafka topics. Unlike ad-hoc queries, which are executed once and then discarded, persistent queries are stored and executed continuously, providing a real-time view of the data.The results of this query are written back to a Kafka topic which we can specify while the query is being created.</p><p>With the help of writing multiple persistent queries, we also have the ability to relay combinations of messages to multiple targets. For eg, I can choose to relay notifications from say source 1 &amp; source 2 to target 1 and source 2 to target 3 by changing our persistent queries accordingly.</p><p><strong>Persistence Layer</strong></p><p>It’s reasonable to ask why can’t I directly read from the topic using consumer groups instead of persisting it in a SQL-based store.</p><p>The main reason for us to choose a SQL store is the requirement for advanced querying capabilities over the data which may not be directly possible from a Kafka topic. One such use-case is retrieving the messages by descending order of the arrival timestamps, and pagination. We also wanted to use the indexing features available only in SQL Databases.</p><p>We have used a Spanner sink connector which is available along with a confluent cloud that can write the messages into Spanner in batches. We used a sink connector to avoid writing complex consumer group code, maintaining connections to Kafka and Spanner thereby fast-pacing the development.</p><p>Sink connector also can handle schema evolution like column addition in the target database. Sink connector writes to the spanner database in batches is one of the major bottlenecks in the systems. We had to carefully optimise for batch size to get the optimal results. We are always looking for ways to compress messages within the batch to reduce the number of writes to Spanner. Messages within a batch are optimised so that only the latest value of the primary key will be written to the target.</p><p>In order to increase parallelism and thereby throughput we can control the “TASKS” parameter which we can imagine as a number of parallel workers available for processing your messages in the topic.</p><p>The messages are “UPSERTED” into the Spanner database so that only one row exists for each notification.</p><p><strong>Notification Querying</strong></p><p>Now the messages are in the Spanner Database. We use the Chronosphere backend (Java Spring boot) to pull the messages designated to a particular user from the database.</p><p>We have carefully chosen the Primary and Secondary Indexes to speed up our queries and avoid any potential hotspots.</p><p>We are also working on auto-expiry of messages based on commit timestamp(supported by Spanner) so that we only have a history of months for a particular user.</p><p><strong>Monitoring and Observability</strong></p><p>We’ve implemented a Grafana Prometheus dashboard to provide real-time insights into our system’s performance. It tracks key metrics like QPS, error rates, and read/write latencies, allowing us to identify and address potential bottlenecks promptly.</p><p>We also monitor closely the Confluent dashboards and look for various parameters such as the number of events ingested into the queue, the number of successful writes by the Sink Connector, messages in backlog, messages in Dead Letter Queue(the mechanism used to handle messages that cannot be processed successfully)</p><p><strong>Conclusion</strong></p><p>I have abstracted many things either due to privacy or for keeping this article shorter. There are a few more things that I would like to address in a separate blog like the Celebrity problem (When a user having large followers creates a post, notifications must be sent to all followers), performance benchmarking etc.</p><p>Change is inherent with every system design and we hope this system is adaptable and can handle our growing userbase and number of notifications.</p><p>Almost every member of our gaming backend team has contributed to this project at some point in time. So would like to thank each one of them. Feel free to connect me on LinkedIn <a href="https://www.linkedin.com/in/vipin-thazhe-madam-474a7814/">https://www.linkedin.com/in/vipin-thazhe-madam-474a7814/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d05499220126" width="1" height="1" alt=""><hr><p><a href="https://engg.glance.com/notifications-system-using-kafka-and-ksql-d05499220126">Notifications system using Kafka and KSQL</a> was originally published in <a href="https://engg.glance.com/">Glance</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>