Software Engineering Dose #5
Hey folks, Back with another edition of Software Engineering Dose. In this edition we will be discussing:
- Scoring and Ranking leaguges Games24x7 Engineering
- Migrating chat data with no downtime
- Redis Pipelining saved 90% of time during writes by SwiggyBytes
- RCA: Injecting metadata as comments into SQL queries by Adyen
Before we go into details, sharing the link to last dose.
Scoring and Ranking leagues @ Games24x7 Engineering
Games24x7 Engineering shares a sneak peek at the Score update and Leaderboard calculation flow for their contests.
It was very interesting why they didn’t choose Redis Sorted Set which is pretty common when calculating Leaderboards but it just wasn’t scalable to them.
Since this problem is already solved(Top k heavy hitters), they just used a data processing pipeline to calculate the leaderboard which is pretty fast.
Although some data on why not could have been more interesting but nevertheless a very interesting read.
2. Migrating Chats data with no downtime
In this article, Beekeeper shares how they migrated their chat data with no downtime.
Since their Chats v1 was kind of outdated and they use HTTP Long polling and they want to move to a Websocket model. For their load requirements, they went with an open-sourced MongooseIM which uses XMPP.
XMPP stands for Extensible Messaging Presence Protocol. It streams XML elements over a network to share messages and presence information. WhatsApp uses a customized XMPP.
This migration is shared in 2 parts. In the first part, they share the producer model wherein entity data is streamed to Kafka and how they did it without affecting live production traffic. Pros and cons were shared on Direct Select Query and Publish vs CDC events.
In the second part, it was mostly the consumer side of it(figure below), converting the entity data received into a compatible format for the ChatsV2 and directly writing to the new DBs bypassing the business layer as it is faster.
Finally, they shared detailed steps and some numbers and the infra they used.
3. Redis Pipelining saved 90% of time during writes by SwiggyBytes
SwiggyBytes shares how as part of infra-cost optimization, were able to save 90% of time in writing to Redis.
Get Sriteja Parimi’s stories in your inbox
Join Medium for free to get updates from this writer.
For this, they used the concept of Redis pipelining which is basically a grouping of commands and sending them all at once.
Here, we save the Round trip time(RTT) for the number of commands we include in 1 batch.
In addition to this, since Redis is single threaded when it comes to Command execution, each command involves a switch between user model and kernel mode for the command to be executed, batching them in 1 also reduces those context switches to 1. This reduced CPU utilisation of Redis.
These are the end results after these changes.
4. RCA: Injecting metadata as comments into SQL queries by Adyen
In this interesting RCA, Adyen shares us their investigation into how adding comments to a query ended up increasing the CPU usage of the RDS.
We have heard about traceId being used extensively to trace a request across different services and any upstream calls as well. So, here they thought of adding some metadata about the query as comments and Postgres ignores the comment part anyway so figured it would help in identifying the troublesome queries and taking action.
But, why stop with static metadata when we can pass anything as comments inside the query? So they added traceId also as a comment to the query. Very interesting thought I must say.
What they saw was a huge increase in CPU utilization on Database servers. Why is it happening?
Well well well, they use JDBC to communicate with DB servers and Prepare statements to execute queries. The advantage with them being that with Prepared statement, the query is divided into 2 parts
1. Parse, Analyse, Rewrite, and Plan
2. Execute
So, when the same query is issued again and again and it exceeds a certain threshold, it gets saved as a Named statement and from next time on, the parameters we pass will be sent to DB and not the whole query and it directly gets executed.
So, this Named statement is like a HashMap let’s say and the key is the query. Anything yet?
So, earlier when we were sending the query, the key value is always the same as we are sending the same query.
But now, since we added some dynamic content as metadata which is traceId , it is being parsed differently and every time, it has to do all the steps again. Well, that caused the spike and so they rolled back on adding traceId as a comment inside the query. 😌
That’s it for this time folks. I really enjoyed reading these articles and preparing notes and hope you find these articles knowledgeable.
Please also share any feedback on how this can be made even better.
Thanks for reading!







