Introduction

At Games24x7, we deliver seamless, high-performance gaming experiences for millions of players, making system performance and scalability our top priorities. For many years, our backend was built on Java 8. However, as technology evolved, we recognized the necessity to upgrade. Java 21 introduced powerful new features and long-term support, marking the ideal moment for this transition. In this blog, we share our real-world migration journey from Java 8 to Java 21, detailing the challenges we overcame, the solutions that succeeded, and the improvements achieved.

If your applications are still running on Java 8, you’re not just missing out on cleaner syntax and enhanced performance, you’re also accumulating technical debt, exposing yourself to potential security risks, and facing increasing incompatibility with modern libraries and frameworks.

In this guide, we will walk you through how we migrated our large-scale production systems from Java 8 to Java 21, detailing what broke, what worked, and what we learned. Whether you’re planning a migration or are simply curious about the changes, this blog will provide you with a roadmap filled with real-world insights.

🚀 Why Migrate to Java 21?

With evolving technology and growing player demands, staying on Java 8 was no longer sustainable. Java 21 brought us the opportunity to modernize our stack, eliminate legacy constraints, and unlock powerful features like virtual threads, enhanced garbage collection, and long-term support. This migration wasn’t just an upgrade — it was a strategic step toward faster performance, tighter security, and future-ready scalability.

⚠️ Challenges with Java 8:

Remaining on Java 8 presented considerable risks to our operations:

• End of Support
  Java 8 has reached the end of public updates under the OpenJDK project, and Oracle’s commercial support is ended. Continuing on this older version left our systems at risk of missing important security patches and performance improvements. Upgrading was essential to ensure long-term stability and maintainability.
• Security Vulnerabilities
  Java 8 lacks modern cryptographic algorithms and misses out on critical security patches provided in newer versions. This increases the attack surface and poses a serious risk to player data and financial transactions.
• Framework Incompatibilities
  Modern frameworks like Spring Boot 3.x, Jakarta EE, and Jetty 12 require Java 17 or above. This made it impossible to adopt essential tools for performance, scalability, and observability while staying on Java 8.
• Performance Limitations
  Garbage collectors in Java 8 (like CMS and early G1) are not optimized for today’s low-latency, high-throughput workloads. This impacted our ability to scale reliably during peak traffic.
• Technical Debt
  Maintaining legacy Java 8 code slows down developer velocity, increases boilerplate, and blocks adoption of cleaner, safer language features. This led to higher maintenance costs and slower feature delivery.

✅ Benefits of Migrating to Java 21:

Java 21 solves these limitations while opening the door to modern, high-performance engineering:

• Long-Term Support (LTS)
  With LTS support, Java 21 ensures consistent updates, stability, and security, critical for our production systems and compliance requirements.
• Enhanced InfoSec Compliance
  Transitioning to Java 21 allows us to eliminate vulnerabilities that were unresolvable within our outdated technology stack.
• Adoption of Modern Technology
  Earlier Java versions restricted our ability to utilize advanced, high-performance tech stacks such as Jetty 12, Jakarta Servlet APIs, and Virtual Threads.
• Advanced Garbage Collectors
  The introduction of new garbage collectors (ZGC and Shenandoah) are optimized for low-pause, high-throughput workloads, resulting in improved startup times, reduced memory usage, and optimized overall performance.
• Virtual Threads (Project Loom)
  By replacing traditional thread pools, virtual threads empower us to manage thousands of concurrent player connections with minimal overhead, significantly enhancing server scalability.
• Modern Java Features for Cleaner Code
  Java 21 offers developer-friendly features like var, records, text blocks, sealed classes, and pattern matching, which contribute to making code more readable, concise, and maintainable.
• Native Image Support via GraalVM
  Java 21 improves integration with native-image builds, facilitating ultra-fast startup times and reduced memory consumption for microservices.

🔍 Java 8 vs Java 21: Feature Comparison

Press enter or click to view image in full size

🧭 Preparation and Planning

Our migration journey began with a detailed audit of our codebase to assess Java 21 compatibility. Key preparation steps included:

• Internal Library Analysis
  We reviewed all internal libraries for Java 21 compatibility. Special attention was given to usages of javax.*, since some annotations may compile without errors but silently fail at runtime.
• Build Tool Updates
  We upgraded Maven to version 3.10.x to ensure Java 21 support. Below is a sample maven-compiler-plugin configuration we used:

<plugin>
 <groupId>org.apache.maven.plugins</groupId>
 <artifactId>maven-compiler-plugin</artifactId>
 <version>3.11.0</version>
 <configuration>
 <source>21</source>
 <target>21</target>
 </configuration>
</plugin>

• CI/CD Pipeline Configuration
  We modified our Jenkins pipelines to test builds against Adoptium’s Java 21 JDK, validating compatibility in isolated environments using Docker containers.
• Version Control
  All migration-related changes were isolated in a dedicated Git branch, allowing for safe experimentation, and easy rollback if necessary.
• Created New Infrastructure
  To minimize risk, we deployed Java 21 builds to a separate infrastructure. This allowed us to safely route live traffic back to the Java 8 environment in case of issues, ensuring zero downtime during transition.

These steps ensured our codebase was ready for Java 21 without disrupting our high-concurrency gaming workloads.

🎯 Migration Approach

Given the scale and complexity of our codebase, we followed a phased migration strategy to ensure stability and minimize risk.

1. Library Compatibility Audit
   We analyzed all external libraries for Java 21 compatibility. For outdated or unsupported dependencies, we referred to official migration documentation and upgraded them selectively, prioritizing business-critical components.
2. Internal library Upgrade
   Updated internal library to make it compatible with java 21. One major change involved migrating from javax.* to jakarta.*, especially to align with Spring Boot 3.x and Jakarta EE 10+.
3. Build Compatibility
   Initially, we focused on ensuring that services built and ran correctly on Java 21 without adopting any new Java 21 language features. New feature we integrated separately.
4. Automated & Regression Testing
   We ran extensive unit, integration, and end-to-end test suites to verify functionality and catch regressions early.
5. Load Testing & Benchmarking
   Performance testing was conducted under production-simulated loads to compare Java 8 vs Java 21 metrics, including memory usage, GC behavior, latency and verified there is no memory leak issue.

🧩 Dependency Compatibility for Java 21 & Spring Boot 3.x

As part of the migration, we upgraded all unsupported third-party and internal dependencies to ensure seamless compatibility with Java 21 and Spring Boot 3.x:

Press enter or click to view image in full size

🛠️ Challenges Faced and Solutions Implemented

Upgrading a large-scale production system from Java 8 to Java 21 came with lots of challenges. Here’s how we tackled them:

• Jetty 12 Migration
  Jetty 12 introduced breaking changes in its WebSocket API. We restructured our WebSocket handling logic to align with the new server model, ensuring seamless bidirectional communication.
• Jakarta Namespace Transition
  The shift from javax.* to jakarta.* necessitated extensive refactoring of our codebase, updating Spring Boot configurations, and upgrading all dependent internal libraries to support Jakarta EE.
• Internal Library Compatibility
  Some internal and third-party libraries relied on deprecated or restricted internal Java APIs. We resolved this by either upgrading to compatible versions or refactoring the code to use supported public APIs.
• Akka Version Conflict
  The incompatibility between Akka 2.12 and 2.16 made it hard to deploy services independently. We resolved this by decoupling Akka-based communication between services to avoid taking downtime.
• MongoDB Server Compatibility
  The new MongoDB Java driver required a minimum server version of 4.0. To avoid mismatches, we upgraded our MongoDB clusters in both staging and production.
• Swagger API Deprecation
  springfox-swagger was incompatible with Java 21 and Spring Boot 3.x. We migrated to springdoc-openapi, which fully supports the new ecosystem while offering equivalent functionality.

📝 Code Transformations

While migrating to Java 21, we had the opportunity to modernize parts of our codebase using new language features. These changes are optional, older Java 8-style code still works but adopting them can lead to cleaner, more expressive code.

Example 1: var usage (Java 10)

// Before
Map<String, String> config = new HashMap<>();
// After
var config = new HashMap<String, String>();

Example 2: Switch Expressions (Java 14)

// Before
switch (status) {
 case SUCCESS:
 result = "OK";
 break;
 case ERROR:
 result = "FAIL";
 break;
}
// After
result = switch (status) {
 case SUCCESS -> "OK";
 case ERROR -> "FAIL";
};

Example 3: Records for Immutable Data Classes (Java 16)

// Before
public class User {
 private final String name;
 private final int age;
 // constructor, getters, equals, hashCode, toString
}
// After
public record User(String name, int age) {}

Example 4: Pattern Matching for instanceof (Java 16)

// Before
if (animal instanceof Cat) {
 Cat cat = (Cat) animal;
 cat.meow();
}
// After
if (animal instanceof Cat cat) {
 cat.meow();
}

📈 Improvements Observed

As part of our migration to Java 21, we observed significant enhancements across performance, security, and code maintainability:

• Performance Gains
  We have observed a 40–80% reduction in latencies across various scenarios. For instance, the FMG search response time has improved by approximately 50% (p99 reduced from 200 ms to 80–90 ms). Graphs are available for reference.
• Vulnerability Reduced
  We successfully reduced over 2500 vulnerabilities, including 180 criticaland 1200 high severity issues across our services. This was achieved by upgrading dependencies and adopting Java 21
• Cleaner, More Maintainable Code
  The new language features have encouraged the development of more maintainable and readable code.

Performance Metrics

To validate the impact of our migration, we benchmarked key services before and after upgrading to Java 21. Below are some notable improvements

Press enter or click to view image in full size

Press enter or click to view image in full size

Press enter or click to view image in full size

💡Tips for Others Planning Similar Migrations

• Start Small: Begin with utility modules or internal libraries before touching core services. This reduces blast radius and builds confidence.
• Invest in Testing: A robust test suite (unit, integration, load) is essential to catch regressions early and validate performance improvements.
• Upgrade Dependencies Early: Identify and upgrade incompatible libraries up front. This clears the path for Java 21 compatibility and avoids obstacles mid-way.
• Educate the Team: Conduct training sessions to familiarize the team with new Java 21 features. Also, communicate key migration changes clearly to avoid incompatible code in future development and reduce bugs.
• Document the Process: Keep detailed records of the migration steps and decisions made to help future migrations and onboarding of new team members.

Conclusion

Migrating from Java 8 to Java 21 has been a significant milestone for us at Games24x7. This upgrade unlocked modern language features, stronger security, better performance, and access to new concurrency models like virtual threads — all critical for running high-scale gaming systems.

Looking ahead, we’re excited to further adopt Java 21 capabilities such as records, virtual threads, and advanced garbage collectors to boost developer productivity and system efficiency.

If your systems are still running on Java 8, now is the right time to migrate. The Java ecosystem has evolved, tooling support is mature, and the long-term benefits in performance, maintainability, and security significantly outweigh the migration effort.

References

https://doc.akka.io/docs/akka/2.5/project/migration-guide-2.4.x-2.5.x.html

https://doc.akka.io/docs/akka/2.6/project/migration-guide-2.5.x-2.6.x.html

https://doc.akka.io/japi/akka/2.4/deprecated-list.html

https://doc.akka.io/japi/akka/2.5/deprecated-list.html

https://github.com/spring-projects/spring-boot/releases/tag/v3.2.1

https://jetty.org/docs/jetty/12/programming-guide/migration/11-to-12.html

https://jetty.org/docs/jetty/12/programming-guide/migration/94-to-10.html

https://jetty.org/docs/jetty/12/programming-guide/server/websocket.html

https://thorben-janssen.com/migrating-to-hibernate-6/

https://www.oracle.com/news/announcement/ocw-oracle-releases-java-21-2023-09-19/?utm_source=chatgpt.com

https://docs.oracle.com/en/java/javase/21/core/virtual-threads.html

https://springdoc.org/migrating-from-springfox.html?utm_source=chatgpt.com

About the Author

Satyam Jaiswal is a Software Engineer at Games24x7, with 3 years of experience in backend development. He is currently part of the RummyCircle backend team, where he works on building scalable, high-performance systems. Satyam is an alumnus of NIT Raipur, where he has completed his B.Tech in Information Technology. He enjoys solving real-world engineering challenges and contributes to delivering seamless gaming experiences at scale.

At Games24x7, operating around the clock is the baseline, not the goal. The real challenge — and the mark of success — is achieving massive scale and unwavering reliability without letting operational costs spiral out of control. While many solutions focus on adding more hardware, the secret to true, sustainable efficiency is often hidden deeper in the architecture. This post is a living example of that principle, showcasing how one of the most fundamental choices an engineer can make — the selection of the right data structure — can become the cornerstone of a cost-effective and highly responsive system.

In this deep dive, we will pull back the curtain on a critical architectural decision that has had a profound impact on our operations. We’ll start by outlining the specific challenges we faced with tracking millions of concurrent users and their activities. We will then compare the data structures we evaluated, explaining why a particular choice proved to be a game-changer. Finally, we’ll share the tangible results, demonstrating how this single decision led to a significant reduction in memory footprint, lower infrastructure costs, and faster performance, proving that the smartest architecture is the key to winning at scale.

Understanding Custom Visible Tournaments (CVTs): Targeted Gaming Experiences

In the highly competitive gaming industry, personalisation is key. Custom Visible Tournaments (CVTs) enable gaming companies to create tournaments and campaigns specifically tailored to target distinct player cohorts, offering gaming experiences that resonate with diverse player bases. For example, a CVT could target high-value players with exclusive tournaments or introduce beginner-friendly competitions to onboard new users.

Diving Deeper into Custom Visibility and Dynamic Entry Fees

Beyond just targeting broad player segments, Custom Visible Contests/Tournaments empower platforms to tailor the user experience at a granular level. Imagine showing specific games only to players from a particular state due to local regulations, or offering exclusive, high-stakes challenges solely to your most engaged users. This isn’t just about showing a game; it’s about reducing clutter and presenting only the most relevant opportunities to each player, dramatically enhancing their engagement.

Taking this personalisation further, Dynamic Entry Fees enable the adjustment of contest offerings based on a user’s financial profile. This means high-rollers might exclusively see high-entry, high-reward tournaments, while players who prefer lower stakes are presented with more accessible, medium-prize alternatives that match their comfort level. Moreover, leveraging a user’s current wallet balance ensures they only see contests they can genuinely afford to join, eliminating frustration and creating a smoother, more enjoyable player journey. This intelligent filtering and fee adjustment capability is crucial for maximising participation and fostering long-term player retention across diverse user segments.

However, as user base grow and campaigns become more ambitious, managing the sheer volume of data associated with these tournaments can become a logistical and technical challenge. With millions of users and thousands of tournaments in play, optimising how this data is stored and accessed becomes a mission-critical task.

This blog explores how we revamped our CVT architecture to solve storage issues, improve performance, and cut costs by leveraging bitmap data structures — a solution that reduced storage needs by 1000x while ensuring scalability.

Current Architecture for our system:

Challenges We Faced in Storing User Data in CVTs

The original CVT architecture was plagued by redundancy and inefficiency. To illustrate the scale of the problem:

Repetitive Data Storage: For every tournament in a campaign, a separate list of user IDs was created and stored in Redis. In scenarios where a campaign included 1,000 tournaments targeting the same 10 million users, this led to the creation of 1,000 individual sets containing identical user data. The resulting overhead was immense, consuming hundreds of gigabytes of memory unnecessarily.

Performance Bottlenecks: Redis operations, such as searching or updating user participation in these tournaments, were slow and resource-intensive. Checking whether a user belonged to a tournament involved scanning large sets, leading to increased latency as the dataset grew.

Operational Challenges and Business: For large campaigns spanning weeks, these inefficiencies not only caused Out of Memory (OOM) errors but also significantly increased the cost of infrastructure due to the need for additional resources.

Team Level Challenges: When the system ran out of memory, our Engineering Team had to wake up in the middle of the night to identify and manually delete big targeted campaigns. This process was sometimes error-prone.

Operational Team Challenges: Due to these issues, the Operational team could not create campaigns longer than two days. They spent a lot of time on operational work, sometimes even during weekends, instead of focusing on strategic initiatives.

Clearly, a fundamental redesign was needed to address these storage and performance challenges.

Guiding Principles for the New Architecture:

While designing the new architecture, one of our guiding principles was meeting the growing demand of the business. When looking for a solution, we wanted the storage cost to be independent of the growing demand of the business and user base. We also wanted to ensure that our Service Level Agreements (SLAs) remained the same or improved compared to the older system.

Solutions Evaluated:

Any disk-based databases like MongoDB:

We explored using a disk-based database like MongoDB, keeping cost in mind. However, we faced latency issues with MongoDB at this humangous data scale. Both insert and get operations were taking O(n) time. As for every Campaign ID, we needed to store all user IDs in a new document. With growing data, we would also need to scale disk/MongoDB clusters continuously. Therefore, MongoDB did not meet our guiding principle for the new architecture, specifically the need for fast, O(1) operations.

Went back to drawing board:

Then we went back to the drawing board and asked ourselves a few key questions:

1. What kind of access patterns do we truly need?
2. If we want to create a solution where storage cost doesn’t increase with growing business, we need a solution where we can’t store the user IDs themselves.

At this point, we asked ourselves:

• Do we truly need to store individual User IDs?
• Our core use case is simply to determine if a given CVT is available for a specific player (i.e., a boolean check).
• This implied that if we could somehow store a Boolean value for each User ID (indicating their inclusion in a campaign), that would be sufficient.
• We then discussed keeping a Boolean array with a size equal to our user base, where the array index would represent the Player ID, and the Boolean value at that location would tell if the player is part of the campaign or not.
• In this model, updating and getting values would both be O(1) time operations.
• This became a game-changing realisation for us! We then discovered that Redis Bitmaps perform exactly this job efficiently. As we were already using Redis and it’s inherently fast, it became our default choice.

Bitmap to the Rescue: How We Reduced Storage by 1000x

Enter the bitmap — a powerful and efficient data structure in Redis. Unlike traditional storage methods, the size of a bitmap depends on the range of user IDs (i.e., the maximum user ID) rather than the number of users it stores. This makes it an ideal solution for use cases where the user ID space is large but sparsely populated.

Key Benefits of Bitmaps

Storage Efficiency with User ID Ranges:

Instead of creating multiple sets for each tournament, we now store a single bitmap per campaign. Each bit in the bitmap corresponds to a possible user ID, with ‘1’ indicating that the user belongs to the campaign and ‘0’ indicating otherwise.

The size of a bitmap in bytes is calculated as: Bitmap size (bytes)=⌈Max User ID/8⌉

Let’s analyze the storage required for different user ID ranges:

This demonstrates how memory usage scales linearly with the range of user IDs, making bitmaps an incredibly efficient choice for storing data about large user populations.

Direct Storage Comparison: Old vs. New Solution

The impact of this architectural shift on storage efficiency is staggering across various user base scales. Consider a scenario where a campaign targets a significant portion of our user base:

This table vividly illustrates the massive reduction in memory footprint. The new architecture’s memory usage is primarily dictated by the maximum possible user ID your system supports, rather than the sheer number of active users within each individual campaign or tournament. This dramatic improvement ensures scalability and significant cost savings, especially when considering campaigns that encompass hundreds or thousands of tournaments all targeting the same user base.

Lightning-Fast Operations: Bitmaps enable O(1) time complexity for both read and write operations. Checking whether a user ID belongs to a campaign or updating their participation status is instantaneous, even for massive datasets.

Scalability: The size of the bitmap does not depend on the actual number of users but rather on the maximum user ID in the dataset. This ensures that the memory footprint remains predictable and manageable, even as our user base grows to hundreds of millions or billions of users.

Cost Savings and Scalability: The Future of CVTs

Cost Analysis:

Under the old system:

• A Redis cluster of 10 nodes (r5.xlarge, 32GB memory) was required, costing approximately $1,814/month.
• Storage for 10 million users per tournament consumed 500MB, significantly limiting the number of tournaments that could be attached to campaigns.

With the new architecture:

• A smaller Elasticache cluster (cache.m6g.large, 6.38GB memory) meets our needs at just $643/month — a 70% cost reduction.
• Storage for 30 campaigns (with each 10 million users) now requires only ~960MB.

Future Scalability:

• The bitmap solution is future-proof, capable of supporting up to 4.2 billion user IDs with minimal additional memory.
• Marketing teams can now create year-long campaigns with hundreds of tournaments, a capability previously unfeasible due to storage limitations and operational overhead.

Operational Simplicity:

No complex data migration was required for this architectural shift. The new architecture could be deployed during a scheduled downtime window with minimal disruption, demonstrating its elegance and efficiency.

About the author:

Sumit Kumar is a Senior Developer at Games24x7, where he serves as the technical owner of the entire Rummy games ecosystem. He specializes in designing and building highly scalable backend systems, translating complex user journeys into elegant, high-performance architecture. Outside of his technical leadership, Sumit enjoys a good game of cricket and unwinding with true-crime docuseries.

Suraj Sharma is a Senior Software Architect at Games24x7 with over 12 years of experience in designing and building cloud-native, internet-scale software.

At Games24x7, trust isnʼt just a principle — itʼs the foundation of everything we do. Whether you’re strategizing on RummyCircle, crafting a winning team on My11Circle, rolling the dice in Ludo, or sharpening your Poker skills, we’ve got one mission: making every moment thrilling, secure, and seamless.

Behind the scenes, advanced technology fuels the magic, ensuring every interaction on our platform is smooth, reliable, and unforgettable. Hereʼs how we’re building a platform youʼll love and trust.

Trust and Privacy: The Pulse of Games24x7

Trust isnʼt given; itʼs earned. Thatʼs why we go above and beyond to safeguard every aspect of your experience. From encrypting Personally Identifiable Information (PII) to implementing strict access controls, your data is always protected.

Handling millions of transactions daily, we’ve perfected the art of combining scalability with security. This means you never have to choose between a safe experience and top-tier performance. With Games24x7, you get the best of both worlds.

Smooth Sailing from Registration to Gameplay

Registration, Simplified

Signing up on Games24x7 is a breeze. Whether you choose email-password, mobile OTP, or email OTP, you’ll be in the game in no time. Add a personal flair with a unique username — select from options like Ace, Alpha, King, or Clown — or let the system pick one for you.

We take authenticity seriously. Mobile verification ensures every player is real, creating a bot-free, genuine gaming community. From the moment you join, it’s clear this is a platform built for trust, fun, and fair play.

Streamlined and Dynamic KYC Verification

Completing your KYC? Simple and secure. Upload documents or verify via OTP — itʼs all quick and user-friendly. We cross-check details against official databases, guaranteeing accuracy and compliance.

KYC isnʼt just about ticking boxes; itʼs our way of creating a trusted space for genuine competition. Every verified user means fewer fake accounts and more transparent, fraud-free tournaments

Why KYC Matters: Fair Play for All

At Games24x7, KYC isnʼt just about compliance — it’s about creating a trusted environment where everyone can play with confidence. By ensuring every user is verified, we eliminate fake players, keeping real-money tournaments fair, transparent, and free from fraud.

Ensuring Data Integrity and Compliance

Trust goes hand in hand with vigilance. We conduct regular checks to prevent duplication, verify age (ensuring all players meet the 18+ requirement), and adhere to local regulations. Combined with our secure data storage practices, these measures uphold the integrity of our platform, guaranteeing a transparent and legally compliant gaming experience.

With KYC, you’re not just verifying your identity — you’re stepping into a community built on fairness, trust, and a commitment to authentic gameplay.

Fast, Transparent, and Secure Transactions

In the dynamic world of online gaming, speed and reliability are everything. Thatʼs why our financial systems are built to handle transactions with lightning-fast precision. Whether it’s deposits, withdrawals, or contest prizes, every transaction is processed in seconds — because who wants to wait when the fun is just a click away?

Safety You Can Count On

At Games24x7, your transactions are always in safe hands. With state-of-the-art security protocols, advanced encryption, and robust fraud detection systems, we ensure your money is secure every step of the way. So, you can focus on the game without a momentʼs worry.

Transparency That Builds Trust

We believe in keeping things crystal clear. Our transaction history page provides a complete view of every deposit, game table transaction, and withdrawal. No hidden fees, no surprises — just full transparency and a platform you can trust.

With Games24x7, you can rest easy knowing your transactions are not only fast but also secure and transparent, leaving you free to enjoy the thrill of the game.

Cutting-Edge Monitoring for Flawless Operation

Our systems are built for precision:

• Real-Time Settlement Monitoring — No missed transactions
• Continuous Ledger Tracking: Every rupee accounted for.
• Daily Reconciliation Alerts: Issues resolved seamlessly, often before you notice.

Real-Time Notifications: Built on Trust, Designed for You

At Games24x7, trust forms the backbone of everything we do, including how we communicate with you. Our push notifications are more than just timely updates — they reflect our dedication to transparency, dependability, and delivering personalised experiences.

Explore the details in this blog: How Games24x7 Delivers Millions of Push Notifications Using AWS.

Why You Can Rely on Our Notifications:

Precision at Scale: When itʼs time to announce an upcoming tournament or share an exclusive offer, you can count on us. Our robust system delivers up to 14 million notifications within minutes, ensuring you’re always in the know, no matter what.

Tailored Transparency: From updates on your favorite games to reminders about your winnings, each notification is carefully curated to provide information that matters to you — nothing more, nothing less.

Real-Time Reliability: With our real-time alerts, youʼll never miss a critical moment. Whether itʼs game-changing news or a time- sensitive promotion, we ensure every notification reaches you promptly and accurately.

At Games24x7, our advanced notification system is more than just technology — it’s a promise. A promise to keep you engaged, informed, and always ahead, so you can play with confidence and peace of mind.

Because when trust leads the way, the fun never stops.

Is Games24x7 Really Legit? A Behind-the-Scenes Story

Imagine this: You’re sitting with friends, scrolling through the internet, when someone says, “Can you really trust games like RummyCircle or My11Circle? I mean, ₹3 Crore prize pools? Fair gameplay? Sounds too good to be true.”

You pause, curious. It’s a valid question. And this is where the story begins — a story of trust, technology, and the people behind the magic of Games24x7.

Doubt 1: “Is the ₹3 Crore prize on My11Circle even real?”

One of our developers smiles and says, “Absolutely. Let us break it down for you.”

At My11Circle, everything is governed by a transparent system. Our contests, including the ₹3 Crore first prize, operate under publicly available rules that leave no room for doubt. Your teamʼs score is based on the real-world performance of players, and our backend runs a highly secure, scalable architecture that ensures every point is logged accurately in real time.

Hereʼs the deal: Once a match locks, your team is set in stone — no last-minute changes, no hidden adjustments. The leaderboard updates dynamically in real time as points are scored, and prizes are credited to the winners almost instantly after the match concludes.

Discover the full story here: Distributed Scoring and Ranking Through Thor.

To make it even more transparent, participants can download all teams of a contest after the match begins. This feature allows anyone to cross-verify the fairness of results once the match is concluded.

“You can literally watch your rank climb as players perform on the field,” the developer adds. “Itʼs like watching magic, only itʼs powered by math, code, and trust.”

Clearing the Air About Our Fantasy Game Prizes

We understand that big prize pools like ₹3 Crore can raise questions. Is it for real? Are the winners legitimate? Is the prize money actually paid out?

To ensure transparency, we’ve launched our Winners Page, where you can view real winners, their profiles, and prize amounts. Youʼll also find details on the total prize pool, white goods awarded, and nearby winners.

Our goal is to give you confidence in the fairness and authenticity of our contests. By showcasing genuine winners, weʼre building trust and excitement — because when you win, you win for real!

Doubt 2: “What about fairness? How do I know itʼs not rigged?”

Another team member chimes in, “Ever heard of RNG-certified systems?”

For games like RummyCircle and Ludo, randomness isnʼt left to chance — itʼs algorithmic and certified. Our card shuffle systems on RummyCircle are designed to be tamper-proof, ensuring every deal is completely random. And when it comes to Ludo, our Random Number Generator (RNG) ensures that every dice roll is genuinely unpredictable.

“Itʼs like a digital vault,” they explain. “Even though we built the system, we’ve designed it to be completely tamper-proof. Itʼs powered by industry-standard security practices, including recommendations from iTechLabs, a trusted certification authority. Thatʼs how we ensure your data and experience remain secure and reliable.”

And what about suspicious activity? Our fraud detection tools are no joke. Using machine learning and real-time monitoring, we scan every move for patterns of bot usage or collusion. If something doesnʼt look right, itʼs flagged instantly, and our system locks out any fraudulent activity before it impacts the game.

Doubt 3: “Okay, but what about fake players in real-money tournaments?”

This time, a data engineer steps in with a confident smile. “Thatʼs why KYC is a non-negotiable step for us.”

Before you can even join a game, you pass through a rigorous verification process. Mobile number verification, official ID checks, and age validation ensure that every player is real and meets our strict eligibility criteria (yes, we confirm all users are 18+).

“Every player you compete against is genuine,” the engineer assures. “No bots, no fraud — just authentic competition at its finest.”

But we donʼt stop there. Games24x7 has implemented a multi-layered fraud detection framework, as detailed in our Fortress of Fair Play initiative. This initiative uses advanced machine learning algorithms, real-time monitoring, and secure OTP verification to ensure the integrity of every match and transaction.

Through these systems, we can quickly detect and prevent suspicious activities like bot usage, collusion, or duplicate accounts. Our team continuously monitors for anomalies, ensuring that every player competes on a fair and level playing field.

The engineer concludes with a reassuring tone: “Whether itʼs through KYC or cutting-edge fraud prevention, we’re committed to maintaining the fairness and trust that our players deserve. So when you play, you can focus on the thrill of the game — weʼve got the rest covered.

Doubt 4: “Can my transactions really be safe?”

The head of security leans forward with a grin. “You bet they are.”

Every deposit, withdrawal, and prize credit is encrypted using the latest security protocols. Our transaction monitoring system tracks millions of data points daily, flagging anything unusual in real time. And for added transparency, you can view every deposit, game transaction, and withdrawal on your transaction history page — no hidden fees, no surprises.

“Our systems are designed to process withdrawals in seconds,” they add. “Because your winnings are yours, and you shouldnʼt have to wait for them.”

Doubt 5: “But what happens if the system breaks?”

A backend engineer chuckles and responds, “Break? Thatʼs not even in our vocabulary.”

At Games24x7, resilience isnʼt just a feature; itʼs a cornerstone of our platform. Our systems are engineered to handle massive traffic seamlessly, even during the most intense moments.

• Push Notifications at Scale: Need updates? No problem! Our platform can send more than 14 million push notifications within minutes, ensuring every player stays informed, no matter the stakes.
• Contest Joins: Our backend seamlessly manages high-stakes contests, even during peak activity. We’ve processed up to 14 million contest joins with a peak throughput of 4.94K requests per second (RPS) — all without a hitch. Learn more here: Serving Millions of Contest Join Requests — My11Circle.
• Auto-login: When itʼs game time, players flood in to check contests or winnings. Our systems process app openings at a blistering speed of 2.75K RPS, ensuring no delays in getting you into the action.
• Efficient Payouts: The best part? Once the match is over, winnings are credited almost instantly post-settlement. From leaderboard rankings to prize transfers, every process is optimized for speed and accuracy.

How do we make it happen? Our secret sauce includes distributed databases for data integrity, event-driven architectures for lightning-fast response times, and robust failover mechanisms to eliminate downtime. Real-time observability tools continuously monitor everything — from transaction logs to pending settlements — so we catch and resolve issues before you even notice them.

The engineer wraps up with a confident grin: “Whether itʼs push notifications, contest joins, or crediting your winnings, we’ve built our systems to handle the pressure. So you focus on the game — weʼll make sure everything else works like clockwork.”

The Big Picture

As the conversation wraps up, one of the team members leans back and says, “Look, we know trust isnʼt given — itʼs earned. Thatʼs why weʼve built every system, every feature, and every line of code with one goal in mind: to give you a platform where you can play with confidence.”

So the next time someone questions whether Games24x7 is legit, youʼll know the answer. Behind every game, every leaderboard update, and every prize is a team of developers, engineers, and security experts working around the clock to create a platform thatʼs fair, transparent, and built for you.

The only thing left to do? Dive in and see the magic for yourself.

The Games24x7 Promise

At Games24x7, we blend cutting-edge technology with an unwavering commitment to trust and transparency. From seamless registration and quick payouts to secure gameplay, every part of your experience is designed with you in mind.

So, whether you’re aiming for the top of the leaderboard or just enjoying a casual game, you can count on us to deliver a platform thatʼs fair, fun, and built for you.

Ready to play with confidence? Letʼs roll!

About the Authors

Ankit D Sharma is an SDE-2 at Games24x7, part of the Platform team. Passionate about coding challenges, he frequently explores emerging technologies, participates in hackathons, and has a particular affinity for AWS solutions.

Murali Mohanty is an Engineering Manager at Games24x7 with a B.Tech degree from BPUT. He specializes in problem-solving and excels at building and maintaining distributed systems.

Nithish Prabhu, an SDE-3, is a tech enthusiast building scalable systems at Games24x7. With a passion for cutting-edge technologies, he enjoys solving complex software challenges.

Rohit Kakulapati, is an SDE-3 at Games24x7 and an alumnus of the Birla Institute of Technology, Pilani. With expertise in building scalable backend applications, he also enjoys watching football and reading in his free time.

Abstract

In modern software development, ensuring robust and reliable testing is crucial for delivering high-quality products. Traditional manual creation of test cases (TCs) from user stories often becomes time-consuming and error-prone. This paper proposes an innovative solution that leverages Claude Sonnet 3.0 model via AWS Bedrock, a state-of-the-art generative AI model, for the automatic creation of detailed test cases from specification documents. The solution is integrated into an existing self-serve portal, where Quality Assurance (QA) teams can upload user story documents and receive test case mind maps, facilitating better test coverage and faster delivery cycles. We demonstrate how Claude Sonnet model enhances the process through natural language processing (NLP) to generate structured test cases, mapped into mind map format for easy integration into QA workflows. This approach offers a significant leap toward automating and streamlining test case generation, resulting in enhanced testing efficiency and quality.

1. Introduction

In any software development lifecycle, test case generation is essential to ensure that the software behaves as expected under various conditions. Typically, test cases are manually written based on user stories, which describe the features or functionalities of the software in natural language. However, this manual process can be time-consuming, inconsistent, and prone to human error. With the advent of artificial intelligence (AI), there is a growing opportunity to automate repetitive tasks such as test case generation.

This paper discusses a solution we developed at our organization that integrates Claude Sonnet, a powerful generative AI model, to automate the creation of test cases from user stories provided in specification documents. The generated test cases are output in a mind map format, offering QA teams a visual representation that is easy to navigate. The solution leverages the AWS ecosystem, specifically Claude Sonnet 3.0 via AWS Bedrock, to achieve high accuracy and efficiency in test case generation.

2. Problem Statement and Motivation

Quality Assurance (QA) teams play a vital role in maintaining the reliability and performance of software products. However, the creation of test cases — essential for validating each user story — often becomes a major bottleneck. Writing test cases manually is time-consuming, labor-intensive, and susceptible to human error. It also lacks consistency across teams and projects, especially as the complexity of systems grows.

This challenge becomes even more pressing during the Indian Premier League (IPL) season, a critical period for My11Circle, the flagship fantasy sports platform of Games24x7. With millions of users participating in real-time, feature stability and performance are non-negotiable. In IPL 2024 alone, My11Circle saw a 50% spike in team creations within the first two weeks and a similar increase in mega prize winners. Looking ahead to IPL 2025, user activity is expected to reach new heights.

To keep up with this demand, the product and engineering teams must release numerous high-impact and innovative features — often under tight deadlines. QA engineers are then required to review massive specification documents, frequently spanning 100–150 pages, and turn them into comprehensive test cases within a limited time frame. This creates immense pressure, often resulting in missed edge cases and defects slipping into production.

Recognizing this challenge, we asked ourselves a fundamental question:
Why not use AI to assist QA in this era of intelligent automation?

To address these pain points, we’ve begun leveraging Claude Sonnet via AWS Bedrock to automate test case generation. By converting dense product specifications into structured, mind map-based test cases, we aim to:

• Reduce manual effort for QA teams
• Ensure greater consistency and coverage across test scenarios
• Accelerate feature delivery without compromising quality

This initiative not only enhances testing efficiency but also empowers our QA teams to focus on higher-value work, ensuring My11Circle remains reliable and resilient during its most critical business window — the IPL season.

3. System Architecture and Workflow

3.1. Overview of the System

The solution we propose integrates into an existing self-serve portal used by QA teams in our organization. The user journey is as follows:

1. Upload Spec Document: QA teams upload a user story specification document (in .docx format) that contains detailed user stories and associated requirements.
2. Automated Test Case Generation: Claude Sonnet is used to process the user story, extract key details, and generate structured test cases in a predefined JSON format.
3. Mind Map Creation: The generated test cases are then converted into a mind map (saved as a .mup file) which visualizes the relationships between user stories, test cases, and various attributes such as steps, pre-conditions, and expected results.
4. Download Mind Map: Once the process is completed, the mind map is automatically made available for download, providing a comprehensive, easy-to-read structure for the QA team to review and use in testing.

3.2. System Components

1. Frontend (UI/UX):

• A user-friendly card interface on the web portal allows QA teams to upload their .docx specification files.
• After the file is successfully processed, the option to download the generated mind map becomes available.

1. Backend (API Layer):

• File Handling: The backend handles file uploads, validating that the document is in the .docx format and saving it for processing.
• AWS Bedrock Integration: The backend service integrates with Claude Sonnet through a series of APIs to send user story content and receive generated test cases.
• Mind Map Generation: The backend processes the test case data and maps it into a structured mind map format, which is then saved as a .mup file.

1. Claude Sonnet 3.0 :

• Claude Sonnet via AWS Bedrock is utilized to generate detailed test cases from the specification document. We designed a prompt that guides the model to produce the test cases in a strict JSON format that includes the description, pre-conditions, steps, and expected results for each test case.

3.3. Integration Flow

1. Step 1: User Story Extraction: The .docx file is parsed, and relevant user stories are extracted. The content is then provided to Claude Sonnet in a conversational format, where each user story is processed individually.
2. Step 2: Test Case Generation: Claude Sonnet model processes the provided content and generates structured test cases in JSON format. The response follows the predefined structure, which includes a detailed set of steps, pre-conditions, and expected results for each test case.
3. Step 3: Mapping to Mind Map Format: The test case data is converted into a mind map structure that uses nodes and sub-nodes to represent the user stories and their respective test cases. Each test case becomes a node in the mind map, with related attributes like description, steps, and expected results.
4. Step 4: File Generation and Download: Once the mind map is created, it is saved as a .mup file and made available for download via the frontend.

4. Claude Sonnet’s Role in Test Case Generation

The central component in the success of this system is Claude Sonnet model. Claude Sonnet 3.0 is an advanced generative AI model that excels in natural language processing (NLP) and can understand and respond to complex prompts. The model plays the following critical roles:

1. Text Understanding: Model processes each user story and generates logical, structured responses based on the input prompts. This enables the AI to create highly accurate test cases that cover both positive and negative scenarios.
2. Structured Output: The model was trained to strictly follow the provided JSON format, ensuring that the generated test cases are structured in a way that is both easy to process and useful for QA teams.
3. Contextual Reasoning: By breaking down each user story into manageable chunks and providing them one at a time to the model, the system ensures that model can reason through each story more thoroughly, thus improving the quality of the test cases.
4. Scalability and Flexibility: Model’s ability to scale to handle multiple user stories and generate relevant test cases in parallel makes it ideal for integration in large-scale systems.

5. Results and Evaluation

Example :

User story :

As a user, I want an option to stay logged in, so that I don’t have to enter my credentials every time. As a user, I want to be able to reset my password if I forget it, so that I can regain access to my account.

Prompt to Claude Sonnet model :

You are an expert tester. Your task is to create detailed positive and negative test cases in JSON format. We will provide you with various fields and scenarios, and you must respond with test cases in the following structure:

{
  "Test Cases": [
    {
      "Description": "",
      "Pre Condition": "",
      "Steps": "",
      "Expected Result": ""
    },
    {
      "Description": "",
      "Pre Condition": "",
      "Steps": "",
      "Expected Result": ""
    }
  ]
}

As a user, I want an option to stay logged in, so that I don’t have to enter my credentials every time. As a user, I want to be able to reset my password if I forget it, so that I can regain access to my account.

The Model’s Response?

{
  "Test Cases": [
    {
      "Description": "Verify the 'Stay Logged In' option works correctly when selected",
      "Pre Condition": "User has a valid account and is on the login page",
      "Steps": [
        "1. Enter valid username and password",
        "2. Select the 'Stay Logged In' checkbox",
        "3. Click the 'Login' button",
        "4. Close the browser",
        "5. Reopen the browser and navigate to the application"
      ],
      "Expected Result": "User is logged in automatically without needing to enter credentials"
    },
    {
      "Description": "Verify the 'Stay Logged In' option does not keep the user logged in when not selected",
      "Pre Condition": "User has a valid account and is on the login page",
      "Steps": [
        "1. Enter valid username and password",
        "2. Do not select the 'Stay Logged In' checkbox",
        "3. Click the 'Login' button",
        "4. Close the browser",
        "5. Reopen the browser and navigate to the application"
      ],
      "Expected Result": "User is prompted to enter credentials again"
    },
    {
      "Description": "Verify the 'Stay Logged In' option does not work with invalid credentials",
      "Pre Condition": "User is on the login page",
      "Steps": [
        "1. Enter invalid username or password",
        "2. Select the 'Stay Logged In' checkbox",
        "3. Click the 'Login' button"
      ],
      "Expected Result": "User sees an error message indicating invalid credentials and is not logged in"
    },
    {
      "Description": "Verify the password reset option works with a valid email",
      "Pre Condition": "User is on the 'Forgot Password' page",
      "Steps": [
        "1. Enter a registered email address",
        "2. Click the 'Reset Password' button",
        "3. Check the email for a password reset link",
        "4. Click the link and set a new password",
        "5. Log in with the new password"
      ],
      "Expected Result": "User receives an email, resets the password successfully, and logs in with the new password"
    },
    {
      "Description": "Verify the password reset fails for an unregistered email",
      "Pre Condition": "User is on the 'Forgot Password' page",
      "Steps": [
        "1. Enter an unregistered email address",
        "2. Click the 'Reset Password' button"
      ],
      "Expected Result": "User sees an error message indicating the email is not registered"
    },
    {
      "Description": "Verify the password reset link expires after the specified time",
      "Pre Condition": "User has requested a password reset",
      "Steps": [
        "1. Request a password reset",
        "2. Wait for the reset link to expire (e.g., 24 hours)",
        "3. Attempt to use the expired reset link"
      ],
      "Expected Result": "User sees an error message indicating the link has expired and is prompted to request a new reset link"
    },
    {
      "Description": "Verify the password reset process fails if the new password does not meet complexity requirements",
      "Pre Condition": "User has requested a password reset and clicked the reset link",
      "Steps": [
        "1. Enter a new password that does not meet complexity requirements (e.g., too short or lacks special characters)",
        "2. Submit the new password"
      ],
      "Expected Result": "User sees an error message indicating the password does not meet the complexity requirements"
    },
    {
      "Description": "Verify the user is logged out of all devices after resetting the password",
      "Pre Condition": "User is logged in on multiple devices and initiates a password reset",
      "Steps": [
        "1. Reset the password using the 'Forgot Password' option",
        "2. Attempt to use the application on other devices without logging in again"
      ],
      "Expected Result": "User is logged out of all devices and prompted to log in with the new password"
    }
  ]
}

Generated mindmap :

5.1. User Adoption and Feedback

Since deploying this solution, the QA teams have reported significant improvements in both the speed and quality of test case generation. The key benefits highlighted include:

• Reduced Manual Effort: QA engineers no longer need to manually write test cases, which reduces the workload and the likelihood of human error.
• Increased Consistency: Test cases are structured in a standardized format, improving the consistency and completeness of the testing process.
• Faster Test Coverage: The time taken to generate test cases from user stories has been reduced by over 50%, allowing QA teams to begin testing earlier in the development cycle.

5.2. Performance Metrics

We conducted performance evaluations on the solution and observed the following results:

• Processing Time: The time required to process a user story specification document and generate the corresponding mind map is under 5 minutes on average.
• Accuracy: The test cases generated by Claude Sonnet 3.0 model adhered to the JSON format with a high degree of accuracy, providing a structured set of test cases that matched the input prompts.

5.3. Scalability

The solution has been tested to handle specifications containing hundreds of user stories, with Claude Sonnet processing each in a scalable and efficient manner. This ensures that the solution can be scaled to larger teams and organizations with minimal performance degradation.

6. Future Work

While the current solution is highly effective, there are several opportunities for improvement and expansion:

1. Integration with Other Tools: Future versions could integrate with test management tools such as Jira or TestRail to automatically create and link test cases within existing test plans.
2. Advanced Test Case Optimization: We could explore using Claude Sonnet model to generate more optimized test cases, focusing on edge cases and specific functional aspects of the user stories.
3. Multilingual Support: Expanding the solution to support multiple languages could make it useful for global teams working with specifications in different languages.

7. Conclusion

This blog presents an innovative solution for automating the generation of test cases using Claude Sonnet model, integrated with a mind map visualization format. The system significantly improves the speed, accuracy, and consistency of the test case generation process, reducing the manual effort involved and enhancing the efficiency of QA teams. By leveraging AI and cloud technologies, this solution streamlines the testing workflow and helps organizations ensure high-quality software releases.

About author :

Nikita brings 9 years of experience in automation testing, with a strong passion for emerging technologies and AI innovations. She actively explores cutting-edge solutions to enhance testing efficiency, streamline processes, and drive continuous improvement across the QA landscape.

TL;DR: Our team at Games24x7 faced a monstrous ANR issue that pushed our app’s Google Play Console metrics into the “bad behaviour” zone. After weeks of investigation and some “technical wizardry,” we not only crushed the ANR but also tackled a react-native bug specific to Android 12. Here’s how we did it.

The Problem: A mysterious ANR without a clear cause 🤔

It all started with a spike in ANRs in our My11Circle Android app. The situation got so bad that it pushed our metrics into the “bad behaviour” zone on Google Play Console 😱. The first issue? The root cause message was as cryptic as a riddle wrapped in an enigma. The error log read:
android.os.MessageQueue.nativePollOnce

libandroid_runtime.so (android::android_os_MessageQueue_nativePollOnce + 44) (BuildId: cebd92605c5c36dc0013f1ca7b00edbd)
       at android.os.MessageQueue.nativePollOnce(Native method)
       at android.os.MessageQueue.next(MessageQueue.java:335)
       at android.os.Looper.loopOnce(Looper.java:186)
       at android.os.Looper.loop(Looper.java:313)
       at android.app.ActivityThread.main(ActivityThread.java:8663)
       at java.lang.reflect.Method.invoke(Native method)
       at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:571)
       at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1135)

I mean, seriously — what does that even mean? 🧐

The Breakthrough: Crashes leading to ANR 💡

Then, after countless hours of stress testing (and probably some coffee-fueled breakdowns ☕💀), we got lucky. While trying to reproduce the ANR, we triggered a crash instead. But wait… instead of crashing as expected, the app entered an ANR state.

BINGO! 🎯 This was a crucial breakthrough. The issue wasn’t just a random crash; it was related to the crash handling mechanism. We were getting closer to the cause… and closer to saving the day (saving the week TBH 😅🙈).

Uncovering the Root Cause: The faulty exception handler 🔧

As we dug deeper, we discovered the villain behind the ANRs: our custom uncaught exception handler. Now, this handler was supposed to be our trusty sidekick 🦸‍♂️ — stepping in whenever things went wrong, catching exceptions, and showing the user an error screen with a helpful message and a “relaunch” button.

In theory, it was brilliant. In practice, not so much. 🤷‍♂️

This was our implementation of exception handler:

Thread.setDefaultUncaughtExceptionHandler(new Thread.UncaughtExceptionHandler() {

          @Override
          public void uncaughtException(Thread thread, Throwable throwable) {

            String stackTraceString = Log.getStackTraceString(throwable);
            callbackHolder.invoke(stackTraceString);
            activity = getCurrentActivity();
            Intent i = new Intent();
            i.setClass(activity, errorIntentTargetClass);
            i.putExtra("stack_trace_string",stackTraceString);
            i.setFlags(Intent.FLAG_ACTIVITY_NEW_TASK);
            activity.startActivity(i);
            activity.finish();
          }
      });

The problem here is that when the app encountered a fatal exception, the handler wasn’t checking the activity state properly, and would try to launch a new activity even if the app was in the middle of a critical state (like destroying the current activity). This caused an ANR because the app couldn’t decide what to do next.

The Fix: Refactoring the exception handler 🔨

With the root cause identified, it was time to put on our problem-solving cap and refactor our handler. We were determined to stop the app from freezing and to handle uncaught exceptions like pros. 💼

Here’s what we did:

1. Refactored the exception handler 🔄

The handler got a little “refactor therapy” 💆‍♂️. We updated the logic to make sure we check the current activity state before launching a new one. And if the handler couldn’t manage the exception, we passed it to the original handler.

With that, the ANRs were reduced dramatically, and the app no longer froze when things went wrong. Victory! 🏆

This is our refactored implementation:

originalHandler = Thread.getDefaultUncaughtExceptionHandler();
Thread.setDefaultUncaughtExceptionHandler(new Thread.UncaughtExceptionHandler() {

          @Override
          public void uncaughtException(Thread thread, Throwable throwable) {

            String stackTraceString = Log.getStackTraceString(throwable);
            callbackHolder.invoke(stackTraceString);
            activity = getCurrentActivity();
            if (activity == null || activity.isFinishing) {
              originalHandler?.uncaughtException(thread, throwable);
              return;
            }
            Intent i = new Intent();
            i.setClass(activity, errorIntentTargetClass);
            i.putExtra("stack_trace_string",stackTraceString);
            i.setFlags(Intent.FLAG_ACTIVITY_NEW_TASK);
            activity.startActivity(i);
            activity.finish();
          }
      });

2. Testing & Validation ✅

We didn’t just stop there. We made sure to stress test the app to simulate crashes and see if the fix worked. We also added logs and breadcrumbs to track issue on Firebase Crashlytics. We wanted to make sure our app was no longer hanging around. 🤸‍♂️

The Aftermath: From ANRs to crashes 💥😱

As the ANRs dwindled, a new problem arose: crash reports started flooding in. It turns out, many of the errors that were causing ANRs were now being logged as crashes. But hey, at least they were being logged now! 💬

This meant we had taken a step forward, but also uncovered another layer of issues. We weren’t done yet.

The React Native Bug: The ConcurrentLinkedQueue conundrum 🧵

As the new crashes rolled in, we discovered that the real culprit was a bug in the React Native Android library that was only causing issues on Android 12 devices. Specifically, the issue was tied to ConcurrentLinkedQueue, a data structure that was being used in the react-native and react-native-reanimated.

The Issue: Dropped items on Android 12 🚨

On Android 12, the ConcurrentLinkedQueue was behaving unpredictably. It would drop items under certain conditions, which would eventually lead to crashes. The behaviour was very hard to reproduce as it was not happening on all the devices and it was a very edge case scenario, making it even harder to track down. 😓

The Solution: Replacing ConcurrentLinkedQueue with LinkedBlockingQueue 🛠️

To fix this, we decided to “queue” up a plan. Our solution? Replace the rogue ConcurrentLinkedQueue with the much more reliable LinkedBlockingQueue. We took the issue into our own hands and built a custom version of react-android with this fix. Talk about “queue-trolling” the problem! 😂

Here’s the plan:

1. Cloning and forking the react native android library:
   We forked the React Native Android repo from GitHub and checked out the commit that matched our version.
2. Implementing the patch:
   Here’s how we made it work:

• Android version check: We added a check to see if the app was running on Android 12 (API level 31). If it was, we used LinkedBlockingQueue instead of ConcurrentLinkedQueue.

private final Queue<UIThreadOperation> mQueue;
String versionAndroid = Build.VERSION.RELEASE;
if (versionAndroid.equals("12")) {
    mQueue = new LinkedBlockingQueue<>();
} else {
    mQueue = new ConcurrentLinkedQueue<>();
}

• Patch implementation: We patched the React Native Android library by updating the source code wherever ConcurrentLinkedQueue was used, making it conditional based on the Android version.

3. Building and hosting the custom library:
We had to set our own little “code bakery” to serve the freshest fix! 🍞. Since React Native uses a separate react-android library (and not the one from node_modules), we built and hosted it on our S3 server. Refer this for building react native from source: Building from source

4. Fixing dependency issues:
The fun didn’t stop there! Even after mentioning our version in dependency it was picking the old version of react-android. Then after some debugging we found that React Native’s gradle.properties file was locking the library to an old version, so we had to override it to point to our patched version. Here’s how we made it “snap” into place:

.
.
diff --git a/node_modules/react-native/ReactAndroid/gradle.properties b/node_modules/react-native/ReactAndroid/gradle.properties
index 2ab4e65..7792733 100644
--- a/node_modules/react-native/ReactAndroid/gradle.properties
+++ b/node_modules/react-native/ReactAndroid/gradle.properties
@@ -1,4 +1,4 @@
-VERSION_NAME=0.71.13
+VERSION_NAME=0.80.11-SNAPSHOT-1
 GROUP=com.facebook.react
 .
 .

5. Patching react-native-reanimated:

Our version of react-native-reanimated was fetching pre-built AAR files, so simply applying the patch wasn’t working. Nothing was coming easy to us — no mercy! 💥 So, we had to dive deeper and make changes to its build.gradle file to enable building from source. Once we got that working, we finally applied the final patch with the ConcurrentQueue fixes and build fixes. It felt like finally getting past the toughest level of a game. 🎮🏆. This is how our reanimated patch looks like:

.
.
.
-def aar = detectAAR(REACT_NATIVE_MINOR_VERSION, JS_RUNTIME)
-boolean BUILD_FROM_SOURCE = shouldBuildFromSource(aar, JS_RUNTIME)
+def aar = null
+boolean BUILD_FROM_SOURCE = true
 .
 .
 .
+import java.util.concurrent.LinkedBlockingQueue;
+import android.util.Log;
+import android.os.Build;
 
 public class NodesManager implements EventDispatcherListener {
 
@@ -110,7 +113,7 @@ public class NodesManager implements EventDispatcherListener {
 
   private RCTEventEmitter mCustomEventHandler;
   private List<OnAnimationFrame> mFrameCallbacks = new ArrayList<>();
-  private ConcurrentLinkedQueue<CopiedEvent> mEventQueue = new ConcurrentLinkedQueue<>();
+  private Queue<CopiedEvent> mEventQueue;
   private boolean mWantRunUpdates;
 
   public double currentFrameTimeMs;
@@ -187,6 +190,14 @@ public class NodesManager implements EventDispatcherListener {
     mUIManager.getEventDispatcher().addListener(this);
 
     mAnimationManager = new AnimationsManager(mContext, mUIImplementation, mUIManager);
+
+    String versionAndroid = Build.VERSION.RELEASE;
+    Log.i("VERSION21", "NodesManager RNANIMATED" + versionAndroid);
+    if (versionAndroid.equals("12")) {
+      mEventQueue = new LinkedBlockingQueue<>();
+    } else {
+      mEventQueue = new ConcurrentLinkedQueue<>();
+    }
   }
.
.
.

Testing the fix 🔍🔨

Testing was a real “hunt the bug” situation! Since this was an elusive issue, we couldn’t just rely on automation tests alone. We needed a backup plan:

• Automation: We ran all the automation suites on our build, performed regression testing, and verified that everything is working. Since this was not an easily reproducible scenario, we could not rely only on these results, as these test suits were working earlier as well. That’s why we had to figure out another approach; let’s see what it was.
• Decompiled APK: We wanted to make sure that the correct version of react-android is being used in the final build and has all our changes for the fix. This is how we did it:

1. Decompiled our APK using apktool, which extracted DEX files from our APK: apktool d -r -s universal.apk
2. Converted these DEX files to JAR using dex2jar tool: d2j-dex2jar universal/classes.dex
3. Explored these JAR files using JD-GUI tool, searching for all the classes in which we made changes and verified that those changes are there.

After testing with both automation and APK decompiling, we were confident the fix was solid. It was like finding the final puzzle piece and slotting it into place. 🧩 With that, we rolled out our next release, and the issue was finally laid to rest. 🛌

The Result: A smoother experience for our users 🎉

After deploying the patch, we saw a huge drop in ANRs. The app was running smoothly, and users on Android 12 experienced fewer crashes. Our android vitals really improved and it felt like we’d finally crossed the finish line. 🏁 Win!

Conclusion: The journey is never really over 🚀

What started as a vague ANR message turned into a full-blown investigation that required a series of fixes, patches, and some caffeine. While we successfully reduced ANRs and solved a major third-party library issue, we’re not stopping here. 🚀

In the next phase, we’ll focus on optimising app start times and squashing performance bottlenecks. Stay tuned for more posts on our journey to make My11Circle even more smooth and seamless. 🏃‍♂️💨

About the Author:

I’m an Android developer with 5+ years of experience wrangling Kotlin, dodging NullPointerExceptions, and surviving Gradle builds with my sanity (mostly) intact. When I’m not building apps, I’m sipping chai and explaining to friends for the hundredth time that I don’t actually fix phones.

📬 Connect with me on LinkedIn

One of the most significant challenges in mobile app development is optimising application size without compromising functionality and UX. In the case of My11Circle, we successfully reduced the app size from 63.8MB to 28MB by employing Dynamic Feature Modules (DFM) for our pre-bundled additional game, RummyCircle. However, this journey was far from straightforward, particularly due to the complexities associated with native shared object (.so) files and the Unity Engine, which does not officially support dynamic features.

This blog post outlines the challenges we faced, the solutions we implemented, and how we achieved this substantial size reduction.

THE IMPACT: 56% Decrease in App Size!

Google Play console download size trend over time

Before Optimization

63.8MB (Single Base Module)

After Optimization

28MB (Base) + ~35MB (Dynamic Feature)

Extracting Unity components for RummyCircle into a Dynamic Feature Module allowed us to significantly shrink the base APK size while maintaining all essential features for the primary game, My11Circle.

Business Impact 🚀

From a business perspective, every MB saved in app size directly boosts key acquisition metrics. In this case, reducing the download size by approximately 37MB led to a significant improvement in acquisition, enhancing user engagement and conversion rates.

Dynamic Feature + Unity and Native Libraries

🤔 Why Isn’t This Simple?

At first glance, you might wonder: “What’s the big deal? Shouldn’t it be easy to create a Dynamic Feature and include the Unity engine in it?”

🚨 Well, NO! 🚨

Problem #1: Unity Does Not Officially Support Dynamic Features

A significant hurdle was the lack of official support for Dynamic Feature Modules from the Unity framework. Here is a link to a Unity forum discussion highlighting that there is no direct implementation for dynamic feature modules:
🔗 Unity Forum Discussion

This resulted in minimal official support, scarce online resources, and numerous dead-end StackOverflow inquiries from previous attempts by other developers. Consequently, we had to manually determine how to dynamically load .so files and assets required by Unity at runtime once the dynamic feature was installed.

Problem #2: Handling Native .SO Files in Dynamic Features

In typical use-case, UnityPlayer automatically loads the required .so files during initialization, so developers don’t have to worry about it.

💥 But in our case, this didn’t work! (First major setback)

The culprit? Split APKs installation path.

When an Android app uses Dynamic Feature Modules containing native libraries, the Android framework splits the APKs during installation. This means:

Native .so files are placed inside a separate ABI-specific split APK.

Post Dynamic-Feature installation these files end up in the /splitcompat folder inside filesDir.

UnityPlayer failed to locate and load them ⚠️

To address this, we had to manually extract and load these libraries at runtime after the dynamic feature module was installed.

SplitInstallHelper was of No Help ❌

Google suggests using SplitInstallHelper to automatically locate and load native libraries in dynamic features. However, in certain Android versions, it completely failed to locate the required .so files.

✅ Solution:
We had to brute force locate these files at runtime and manually load them using SoLoader to ensure stability across all devices.

Inconsistent Installation Paths: BundleTool vs. Play Store Installation ❌

Another major challenge was the difference in installation behaviour between:

• Local builds using bundletool
• Play Store console installation (Internal test track)

The installation path for the split APK (dynamic feature) differed significantly in both cases, leading to build failures when testing via the Play Store console.

✅ Solution:
Since the installed .so file path varied in both scenarios, we had to manually handle .so loading in both scenarios. We had to upload multiple builds on Play Store’s internal test track to ensure that fixes worked across different OS versions and OEMs.

Problem #3: Asset Loading Issue

Another significant issue was that Unity does not recognise the split APK path when loading assets from a dynamic feature. By default, Unity attempts to load assets from the base module, resulting in missing asset errors.

✅ Solution: Overriding getPackageCodePath() in UnityActivity
After extensive research and debugging, we resolved this by overriding the getPackageCodePath() method in UnityActivity. This allowed us to dynamically provide the correct path to Unity, ensuring it accurately loaded assets from the dynamic feature module.

@Override
public String getPackageCodePath() {
    if (isDynamicFeatureInstalled()) {
        return getDynamicFeaturePath();
    }
    return super.getPackageCodePath();
}

Problem #4: Unity Engine Load Error

Another unexpected challenge was missing directories upon installation, which caused Unity to fail at runtime. Specifically, Unity would not load correctly unless certain directories were present and detected by the Unity engine.

While the Unity team has stated that they do not support dynamic features, we persevered and ultimately found a solution! 😏

✅ Solution: Adding an Empty JSON File
Through logcat, we discovered that the Unity engine was unable to load the assets prepacked in Unity AAR. Upon inspecting the app installation folder on the device, we found that the required asset directory was missing.

After further investigation, we placed an empty.json file inside the assets/bin/Data/Managed directory of our base module. By adding this empty file, we prevented the directory from being deleted post-installation, ensuring the availability of Unity assets after the dynamic feature installation. This allowed the Unity engine to locate these assets and load them correctly.

🎯 The Final Outcome

By implementing Dynamic Feature Module, manually managing native libraries, and overriding Unity’s asset loading behavior, we successfully reduced the My11Circle app size from 63.8MB to 28MB 😎

This resulted in:

✔️ A 56% reduction in app size (from 63.8MB → 28MB)
✔️ Improved acquisition & conversion rates
✔️ Faster downloads & updates
✔️ A seamless experience for both My11Circle & RummyCircle

💡 Key Takeaway: Unity and Dynamic Feature Modules aren’t designed to work together — but with the right workarounds, massive app size reductions are possible!

🚀 Have you tried optimizing Unity apps for Android? We’d love to hear your experiences!

About the Author:

Abhinav Pawar has over 10 years of experience building mobile apps across various domains.
He thrives on solving complex problems with innovative, out-of-the-box solutions.

Introduction :

In the fast-paced world of fantasy sports, My11Circle stands out by offering sports fans the thrill of building dream teams and competing for real rewards. But behind the scenes, it’s the robust backend technology that ensures a seamless experience, especially during high-stakes events like the IPL and World Cup.
With millions of users creating teams and joining contests in a single match, keeping leaderboards accurate and fast is no small task. To handle this massive scale, we use Apache Cassandra for storing leaderboard data. However, as contests grew bigger, we faced performance issues that challenged our scaling strategies.

The Fantasy Sports Technical Challenge :

Fantasy sports platforms like My11Circle face unique technical challenges:

• Massive Data Volume: Contests can include millions of participants.
• Prompt Availability: Users expect to see final leaderboards immediately after matches conclude.
• Resource Efficiency: Optimizing database utilization to control infrastructure costs.
• Scaling Headroom: Accommodating growing contest sizes without architecture overhauls.

These requirements demand a robust database solution that can handle the persistence and retrieval of large-scale leaderboard data.

Our Leaderboard Architecture :

At My11Circle, we use a two-tier approach for our leaderboard system:

1. Redis: Serves real-time leaderboard updates during active matches.
2. Apache Cassandra: Persists final leaderboard data once matches are completed.

This blog focuses on the Cassandra persistence layer, which faced scaling challenges as our user base grew.

Understanding Cassandra’s Data Architecture :

To appreciate the optimization we implemented, it’s important to understand how Apache Cassandra organizes and processes data.

Cassandra’s Core Concepts :

Primary Key: Unlike traditional relational databases where a primary key simply identifies a row, in Cassandra, the primary key serves dual purposes:

1. Uniquely identifies a row.
2. Determines data distribution across the cluster.

The Partition Key: This is the first component of the primary key that determines which node in the cluster will store the data. It’s crucial for performance and scalability.

Clustering Key: Additional columns in the primary key that determine the sorting order within a partition.

What is a Partition in Cassandra ?

A partition is a fundamental unit of data storage in Cassandra, containing rows that share the same partition key. Understanding partitions is essential because:

• Partitions determine how data is distributed across nodes
• They directly impact write performance
• Properly sized partitions ensure balanced workload distribution

How Data is Partitioned :

1. Partition Key Selection: The partition key (single column or composite) determines data placement.
2. Hashing Process: Cassandra applies a hash function to the partition key to generate a token.
3. Token Assignment: This token corresponds to a value on the ring. Each node in the cluster is responsible for a range of tokens.
4. Data Distribution: It takes the token value and walks clockwise on the ring and whichever node it meets first, that is the node the data is stored in. This token-based distribution ensures data is evenly spread across the cluster.

The Challenge : Hot Partitions at Scale

My11Circle’s Leaderboard System :

At My11Circle, we run multiple contests within each match. These contests vary in size and prize pools:

• Regular contests: Typically lakhs of participants.
• Mega contests: Can reach 20 million participants during Major sport events.

Each contest has its own leaderboard, which is updated in Redis during live matches. Once a match concludes, we persist the final leaderboard in Cassandra through a Spark-based service.

Our Original Table Design :

In a match, each contest is uniquely identified by a contest_id. When users join a contest, they create teams (each with a team_id) and based on performance, each team receives a rank in the final standings.

• contest_id served as the partition key, determining how data was distributed across the Cassandra cluster
• rank and team_id were clustering keys, organizing data within each partition

This design seemed logical at first grouping all teams from the same contest together. However, it created a critical scaling limitation: all data for a single contest was stored in one partition, regardless of size. For mega contests with millions of participants, this resulted in extremely large partitions that overwhelmed individual Cassandra nodes.

The Problem Emerges :

As our platform grew, we began experiencing performance issues:

Observed Metrics and Bottlenecks

• Uneven Resource Utilization: During high-traffic matches, some nodes were overworked while others remained idle.
• Node CPU Imbalance: We observed significantly higher CPU utilization on specific nodes in our Cassandra cluster, reaching 80% CPU while others remained at 30–40% during the same workload period.

CPU

• Warning Logs: Examined server logs on impacted Cassandra node.
  `WARN [CompactionExecutor] Writing large partition fantasy_keyspace/leaderboard:127937062 (438.982MiB)
• Partition Size Explosion: Mega contests created partitions exceeding 500–800 MB, far beyond Cassandra’s recommended 100 MB limit.

• Data persistence time : Data persistence time increased as load increased.

Diagnosis : Hot Partitions

The root cause became clear: our design was creating “hot partitions” , partitions that received disproportionately high write traffic compared to others. The collected metrics served as evidence of the issue.

Why Partition Size Matters in Cassandra :

Oversized partitions create several critical performance issues in Cassandra:

1. Write Performance Degradation

Large partitions require more resources to process updates, causing increased latency during leaderboard persistence.

2. Memory Pressure

Oversized partitions consume disproportionate amounts of memory, triggering more frequent garbage collection and affecting overall system performance.

3. Compaction Overhead

Cassandra’s compaction process struggles with large partitions, consuming excessive resources and potentially falling behind during peak periods.

4. Node Imbalance

Hot partitions create uneven workload distribution, making some nodes work significantly harder than others in the cluster.

5. Scalability Ceiling

As contest sizes grew with each tournament, we faced an architectural ceiling that couldn’t be efficiently solved through vertical scaling.

Reducing partition size was essential to create a stable, predictable, and cost-efficient database layer for our growing platform.

The Solution : Bucketization for Better Distribution

After analyzing our options, we implemented a distribution technique called “bucketization” to evenly spread data across partitions.

Finding the Optimal Bucket Key :

We needed to identify a bucketization value that would:

• Be readily available during query execution
• Ensure uniform data distribution
• Require minimal application modifications

Since users must create teams to participate in contests, and team rankings are already available pre-query, we leveraged these rankings as our bucket key for optimal data distribution.

Bucketization Implementation :

Rather than relying solely on contest_id as the partition key, we enhanced our approach by incorporating a calculated bucket value:

Bucket Formula: bucket = (rank / K) + 1

Through extensive load testing, we determined K = 10,000 provides ideal performance:

• Teams ranked 1–9,999 are assigned to bucket 1
• Teams ranked 10,000–19,999 are assigned to bucket 2
• This pattern continues for all rankings

New Table Schemas :

Note the composite partition key (contest_id, bucket), which spreads data across multiple partitions.

Key Performance Improvements :

1. Even Resource Distribution: CPU and memory usage became balanced across nodes.
2. Enhanced Scalability: The system can now handle larger contests without expensive hardware upgrades. This has enabled us to scale horizontally, helping us save approximately 50% in costs during peak tournament seasons.

3. Eliminated Hot Partitions: No more partition size warnings or performance bottlenecks.

4. Write Efficiency: The 99% reduction in write latency ensures faster updates.

5. Faster Leaderboard Availability: The 33% improvement in data persistence time ensures the final leaderboard is visible to users sooner, enhancing their experience.

The Impact: Faster, Leaner and More Scalable :

The results were dramatic :

Implementation Strategy : Phased Rollout

To minimize the risk, we implemented a phased rollout strategy:

1. New Table Creation:
   - We created the new leaderboard table in the existing Cassandra cluster.
2. ZooKeeper-Based Configuration:
   - We used ZooKeeper to manage which matches/series would use the new table structure
   - This allowed for gradual transition on a match-by-match basis
   - Common configuration was deployed so multiple services could read from the same path
3. Dual-Write/Read Logic:
   - During settlement, our Spark service (Thor) would check if the matchId/seriesId was in the ZooKeeper config
   - If included, data would be persisted to the new leaderboard table.
   - For reads, our Leaderboard Service would check the same ZooKeeper config to determine which table to query for a particular matchId.
4. Progressive Rollout:
   - Started with small-scale matches first
   - Gradually expanded to larger matches as confidence grew
   - Monitored performance at each stage

This approach allowed us to validate improvements in a controlled manner before full deployment.

Lessons Learned :

This optimization journey taught us several valuable lessons:

1. Data modeling is crucial : In distributed databases, how you structure your data is as important as the infrastructure itself.
2. Anticipate growth : Design for future scale, not just current needs.
3. Monitor partition sizes : Regular monitoring helps catch issues before they become critical.
4. Simple solutions can have big impacts : Sometimes a small change in data structure yields tremendous performance gains.
5. Phased rollouts reduce risk : Gradually implementing changes allows for validation without widespread impact.

Conclusion :

A small but strategic change adding a simple bucket for partitioning made a huge impact on our platform’s performance and scalability. This change helped us avoid expensive vertical scaling and ensured a smoother, faster experience for millions of users during high-stakes events.

The improvement didn’t just enhance technical metrics; it directly impacted user experience by ensuring faster contest leaderboard generation and more reliable service. As we continue to grow, this optimization will enable us to scale efficiently while maintaining the performance our users expect.

It’s a great reminder that sometimes, the simplest solutions bring the biggest results.

About author :

Mayank Sisode is working as a Software Engineer at Games24x7, with around 4.5 years of experience in backend development. He enjoys solving challenging problems and loves working on distributed systems that handle scale and complexity.

References :

1. https://www.instaclustr.com/blog/cassandra-data-partitioning/
2. https://docs.datastax.com/en/cassandra-oss/3.0/cassandra/cassandraAbout.html

At Games24x7, we handle an extremely high throughput during peak fantasy seasons. Most of our user-path services are built in Java, and to maximize CPU utilization at scale, we rely on reactive frameworks for efficient resource management. However, writing reactive code requires specialized expertise, as it follows a fully asynchronous programming model, unlike traditional Java.

In this blog, you’ll gain a practical understanding of Java 21’s virtual threads and how they behave in real-world, high-throughput systems. We’ll walk you through a production issue we faced — where our service became unresponsive due to thread pinning caused by synchronized blocks — and how we investigated, reproduced, and ultimately resolved it.

By the end of this blog, you’ll learn:

• ⚙️ How virtual threads differ from platform threads, and why they’re game-changing for I/O-heavy applications
• 🧵 What thread pinning is, why it happens with synchronized blocks, and how it affects virtual thread performance
• 🔄 How ReentrantLock helps avoid thread pinning, and when to prefer it over synchronized in a virtual thread context
• 🔍 How to debug virtual thread issues using Java Flight Recorder and thread dumps
• 🧪 How to reproduce and simulate thread pinning issues to validate assumptions before making architectural changes
• 🚫 Common pitfalls when migrating to virtual threads, and how to avoid them in production
• ✅ Actionable best practices for safely adopting virtual threads in high-concurrency environments

Let’s get going -

What Are Virtual Threads?

Virtual threads are lightweight threads managed and scheduled entirely by the JVM. Unlike traditional Java threads, which have a 1:1 mapping to OS threads and rely on the operating system for scheduling, virtual threads are decoupled from OS threads, allowing the JVM to efficiently manage them.

This design enables optimized thread scheduling, significantly improving resource utilization and allowing thousands or even millions of virtual threads to run concurrently with minimal overhead

How Virtual Threads Enhance Performance and Scalability

Virtual threads are mounted on traditional platform threads (carrier threads). They are designed to yield control to the scheduler when they encounter blocking operations, such as I/O waits. When a virtual thread blocks, the JVM unmounts it from its carrier thread, freeing up the carrier thread to handle other tasks. This efficient scheduling mechanism ensures better resource utilization and scalability.

While virtual threads offer significant performance benefits, they have certain caveats. We encountered one such challenge in our production systems, which we will explore in the following sections, along with our investigation and resolution.

Facing the Unexpected

The service runs on Java 21 with virtual threads enabled, utilizing 4 CPUs and 12GB of memory per pod.

In our fantasy app, a match refers to a real-world cricket (or other sport) game in which users participate by creating fantasy teams. During peak match times — especially around the match start — we observe a traffic surge from 2,000 RPS to around 10,000–13,000 RPS for this service. This increase, nearly 5x, can occur within just 2–3 minutes, and in some cases, within as little as 30–45 seconds. The elevated traffic persists as users engage with live features and real-time updates.

Although the service was designed to handle such load, we observed that some pods became unresponsive during these spikes, failing to serve requests — including health checks and Prometheus metrics APIs. Eventually, these pods were restarted after three consecutive health check failures, in accordance with the liveness probe configuration.

With no working endpoints during this period, we had no visibility into what was happening, making the issue even more challenging to diagnose.

Tracking the Culprit

We were able to reproduce the issue in our load-testing environment using a similar load pattern and identified one API that was experiencing high latency.

This API fetched data from a local cache (LoadingCache from the Caffeine library) while also executing its business logic. At this point, our investigation focused on the loading cache.

Below is the configuration of our LoadingCache:

LoadingCache<Long, Set<Long>> idVsCtSetLoadingCache;
private void initLoadingCache(){
   idVsCtSetLoadingCache = Caffeine.newBuilder()
           .expireAfterWrite(30,TimeUnit.SECONDS) // expire strategy
           .maximumSize(100)
           .build( id ->{
               try {
                   return makeAnIO(id); // Rest call to another service
               } catch (Exception e) {
                   LOGGER.error("Exception in initLoadingCache loader", e);
                   throw e;
               }
           });
}

Since we were performing I/O operations, we suspected that latency in I/O was causing requests to time out. To test this hypothesis, we modified the cache expiration strategy to RefreshAfterWrite.

To our surprise, the initial requests still failed, but once the cache was populated, the issue disappeared. This was our first breakthrough — LoadingCache was the culprit. But what made it so special?

What’s Special -

As we dug deeper into virtual threads, we discovered a crucial detail — they can cause thread pinning when a synchronized block is used. We will explore thread pinning in more detail later, but this finding was a turning point in our investigation.

Although we weren’t explicitly using synchronized in our application code, we suspected LoadingCache. A closer look at how LoadingCache’s get() method works revealed that it relies on synchronized blocks in multiple places — especially when the cache is empty, being populated, or when an entry expires (with expireAfterWrite configured)

Below is the relevant code inside LoadingCache.get(), which internally calls ConcurrentHashMap’s compute() function::

With this, we were confident that thread pinning was our issue, and we needed to validate it with a simple test.

As mentioned earlier, virtual threads are designed to be unmounted from platform threads when they encounter blocking I/O, allowing the platform thread to continue handling other tasks. However, if a virtual thread is blocked inside a synchronized block, it gets pinned, preventing the JVM from efficiently scheduling other virtual threads. This was precisely the issue we encountered

Understanding Thread Pinning

Thread pinning in Java Virtual Threads occurs when a virtual thread is bound to a platform thread for an extended period, preventing the JVM from efficiently scheduling other virtual threads.

However, problems arise when a virtual thread encounters a blocking operation that does not yield control back to the JVM, such as:

• Native synchronization mechanisms like synchronized blocks.
• Blocking I/O operations such as Synchronous File I/Os, JDBC calls and socket I/Os.

When this happens, the virtual thread gets pinned to the underlying platform thread. Since the platform thread is now occupied, the JVM cannot reuse it to execute other virtual threads, limiting the benefits of virtual threading.

But One Pinned Thread Shouldn’t Be a Big Deal, Right?

At first, this raised a question for us: even if one platform thread is pinned, shouldn’t the other platform threads still handle the load? After all, a few pinned threads shouldn’t bring the entire system down.

This made us dig deeper into what was happening under the hood during thread pinning and why this issue doesn’t occur in traditional Java applications using OS threads.

In the next sections, we’ll uncover the real reason why our service became unresponsive and how virtual threads behave in such scenarios. Stay with us!

Simulating the issue

The first step in our investigation was to reproduce the issue in a controlled local environment using a simple Spring Boot application.

We designed an application with two API endpoints:

1️⃣ Business Logic API — This endpoint simulates real-world processing by executing some business logic and then sleeping for 20 seconds inside a synchronized block.

2️⃣ Health Check API — A lightweight endpoint that simply returns 200 OK, mimicking real health checks used in production.

By running this setup under load, we aimed to observe how virtual threads behave when blocked inside a synchronized block, helping us confirm our suspicions about thread pinning.

Application.yaml settings for enabling virtual threads

spring:
 application:
   name: caffiene-load
 threads:
   virtual:
     enabled: "true"

Business Logic API-

public List<UserDetails> getUsersForMatch(long matchId) throws ExecutionException, InterruptedException {
   List<UserDetails> userList = new ArrayList<>();
   synchronized (LOCK){
       for(long i=0; i< 100; i++){
           UserDetails userDetails = new UserDetails(i, "user"+matchId, "shell");
           userList.add(userDetails);
       }
       Thread.sleep(20000);
   }
   return userList;
}

Health Check API -

public ResponseEntity<?> health() {
   try {
       Thread.sleep(3000);// 3s sleep to track the thread in JFR for longer duration.
   } catch (InterruptedException e) {
       throw new RuntimeException(e);
   }
   return ResponseEntity.ok(Thread.currentThread().getName());
}

Dockerfile -

To get deeper insights into what was happening at the system level, we containerized our test application by creating a Docker image.

Inside the container, we installed:
strace — To trace system calls made by the application.
procps — To monitor process and CPU usage in real time.

FROM openjdk:21-slim
EXPOSE 8080

RUN apt update && apt install procps -y
RUN apt install strace -y
ADD target/test-local-cache-load-1.0-SNAPSHOT.jar /home/tomcat/test-local-cache-load-1.0-SNAPSHOT.jar

ENTRYPOINT ["bin/bash","-c"]
CMD ["java -jar /home/tomcat/test-local-cache-load-1.0-SNAPSHOT.jar"]

docker-compose.yaml to simulate a pod with restricted resources and take thread dumps and JFR (Java Flight Recorder) recordings.

version: '3.8'

services:
 app:
   image: caffiene-load:latest
   container_name: caffeine-load-test
   ports:
     - "8080:8080"
   environment:
     - JAVA_OPTS=-Xms2g -Xmx2g
   deploy:
     resources:
       limits:
         cpus: "2.0"
         memory: "3g"
       reservations:
         cpus: "2.0"
         memory: "3g"
   volumes:
     - /Users/ssg/ssg/thread-dumps:/home/tomcat/thread-dumps

To replicate the issue more closely, we restricted our test environment to 2 CPU cores and followed these steps:

1. Started the application.
2. Called the main API twice, manually hitting it sequentially with a 1–2 second interval.
3. Called the health check API immediately after step 2.

Observation:

The health check API was blocked until the first API call completed (20 seconds) and the thread was freed. This meant that no other requests were processed during this period, including the health check!

This was a clear indication that our platform threads were getting pinned, preventing the JVM from scheduling other virtual threads.

Let’s break down why this happened

What Are My Threads Doing?

To get deeper insights into thread behavior, we started Java Flight Recorder (JFR) for 200 seconds inside the Docker container and re-ran our test scenario from the previous section.

You can use the following command to capture a JFR recording using jcmd:

jcmd <java pid> JFR.start name=myRec filename=<filepath>/<filename>.jfr duration=200s

After 200 seconds, we ran the JDK mission control tool to analyse the JFR recording, and we found the following observations -

​​From our first JFR image, we identified the pinned threads. Let’s break down what we observed:

1. The first thread (tomcat-handler-0) entered the synchronized block at 09:07:23 and slept for 20 seconds. It was freed only at 09:07:43.
2. As soon as the first thread(tomcat-handler-0) exited the synchronized block, the second thread (tomcat-handler-1) acquired the lock and got pinned in the same manner

At this point, we had two important questions:

1. Why were there only 2 platform threads (ForkJoinPool) and not more?
2. If both platform threads were getting pinned sequentially, why didn’t the waiting health check API execute on another thread?

Answering These Questions

1) Why Only 2 Platform Threads?

Since virtual threads are entirely managed by the JVM, the number of platform threads is determined based on available CPU cores.

• In our Docker container, we restricted CPUs to 2, so the JVM allocated only 2 platform threads for the ForkJoinPool.
• This means only two virtual threads could be scheduled at any given time, making thread pinning a critical bottleneck in this case

2) Why Didn’t the Health Check API Execute?

To answer this, let’s look at the second JFR image, which shows a blocked Java monitor (i.e., a blocked thread):

• The second thread (tomcat-handler-1) was waiting to acquire the lock on the synchronized block and was completely blocked.
• Because this thread was blocked, the JVM did not unmount the virtual thread from the platform thread.
• This meant that both available platform threads were pinned, leaving no free threads to execute the health check

The result? A complete service freeze.

This was our End of the World moment — both platform threads were stuck, leading to service unresponsiveness.

If we run the same code with virtual threads disabled and provide tomcat max threads to be more than 2 then we can see that this issue doesn’t come because a new platform thread is created once the earlier 2 threads are blocked on synchronized block (1st inside the block and 2nd waiting for the lock). The health check request is executed on a 3rd thread. Here also, if we limit the Tomcat max threads to 2, then we can also see a similar issue that the health check api is not executed, as both the threads are blocked on a synchronized block.

Let’s dive deeper into the second point to understand why the thread waiting on the synchronized block remains blocked

Demystifying The World of Virtual Threads

Based on our discussion, we’ve pinpointed the issue to the synchronized block — specifically, how the platform thread behaves once it enters the block and how the virtual threads waiting for the lock are affected.

As per Java documentation -

When a virtual thread is inside a synchronized instance method and invokes Object.wait() on the object, the virtual thread blocks in the JVM until awakened with Object.notify() and the carrier re-acquires the monitor. The virtual thread is pinned. It is executing inside a synchronized method, and further pinned because its carrier is blocked in the JVM.

This meant that both our platform threads from the previous discussion were completely occupied. Our curiosity kicked in, and we couldn’t resist diving deeper. So, we set out to prove it ourselves — let’s uncover what’s happening at the OS level

What Is Happening At The OS Level?

We now understand that when platform threads are blocked, the JVM cannot process new requests or perform other tasks using the fork-join pool until those threads are released. Since each platform thread is mapped 1:1 to an OS thread, this also implies that the corresponding OS thread is blocked as well.

To validate this hypothesis, we decided to trace all futex (fast user-space mutex) syscalls at the OS level using strace. To capture futex activity, we ran the following commands

nohup strace -f -tt -p <Java Pid> -e futex > strace.out 2>&1 &
ps -T -p <Java Pid>

The first command traces all futex syscalls made by the Java application at the OS level. The second command helps identify the PIDs of fork-join pool threads — these threads are created only when an API is hit or a task is running on them. So, to capture the correct data, we first hit the APIs and then ran the second command to get the PIDs

In our case, after hitting two APIs followed by the health check API, we checked the fork-join pool PIDs using the second command. We observed that both fork-join pool threads were created. Next, we filtered the futex traces specifically for these threads, and here’s what we found.

Thread with the first request

This is the 1st thread that takes the lock and sleeps for 20 seconds in the synchronized block

Thread with the 2nd request

The above image shows the other thread, which tries to take the lock, and that goes on for 20 seconds, and every request is timing out as the lock is not available. This is all happening at the OS level, which shows that the control is now with the OS.

The Saviour — Reentrant Lock

So far, we’ve discussed how the synchronized block leads to thread pinning, causing waiting threads to be blocked. To avoid this, Java documentation suggests using ReentrantLock instead.

Naturally, we wanted to test this ourselves. We modified our previous method, replacing the synchronized block with ReentrantLock. Below is the updated method

private ReentrantLock reLock = new ReentrantLock();
public List<UserDetails> getUsersForMatch(long matchId) throws Exception {
   List<UserDetails> userList = new ArrayList<>();
   try{
       reLock.lock();
       for(long i=0; i < 100; i++){
           UserDetails userDetails = new UserDetails(i, "user"+matchId, "shell");
           userList.add(userDetails);
       }
       Thread.sleep(20000);
   } finally {
       reLock.unlock();
   }
   return userSet;
}

After replacing the synchronized block with ReentrantLock, we repeated the same sequence of steps and generated a new JFR (Java Flight Recorder) file. Here’s what we found

• Health Check API Execution:
  - Unlike before, the health check API was executed immediately and was not blocked.
  - Initially, we assumed that the first thread might still be blocked inside the lock, but the waiting thread was free and handled the health check request.
• Fork-Join Pool Behavior:
  - We triggered the Business logic API twice sequentially, meaning one request was inside the lock, and the second request was waiting.
  - Immediately after, we checked the fork-join pool PIDs using: ps -T -p <Java Pid>
  - To our surprise, only one active thread was created!
  - The second thread was not even spawned in the fork-join pool.

This was an unexpected but crucial finding. Instead of pinning and blocking additional threads, ReentrantLock prevented unnecessary thread creation, ensuring efficient thread management. But why did this happen? Let’s dive deeper.

Let’s break down the last observation -

• First API Call Execution:
  - A platform thread from the fork-join pool is assigned to a virtual thread.
  - The virtual thread enters the ReentrantLock and sleeps for 20 seconds.
• Second API Call Execution:
  - When the second request arrives, the same platform thread is reused.
  - This happens because the virtual thread inside the ReentrantLock(reLock) gets unmounted, freeing the platform thread.
  - The new request gets assigned to a new virtual thread, which runs on the same freed platform thread.
• Health Check API Execution:
  - When we hit the health check API, the same freed platform thread is used to process the request because, unlike the synchronized block, where a waiting thread remains blocked, here the waiting virtual thread unmounts, keeping the platform thread free for other tasks.
• Final State (After Sleep Completion):
  - Once the first request’s sleep completes, its virtual thread needs a platform thread to resume execution.
  - Similarly, the second virtual thread, which was waiting for the lock, now acquires the lock and also needs a platform thread.
  - At this point, two platform threads are required to complete both pending tasks.
  - To verify this, we ran the ps -T -p <Java Pid> command as soon as the first request completed.
  - We observed that two fork-join pool threads were now active, confirming that platform threads are efficiently reused rather than being blocked.

The above observations are exactly what is expected from a virtual thread’s behaviour to make the system more efficient and performant. Below is the image showing no pinning and no blocked threads in the JDK mission control tool for the JFR file -

So now the question arises, what’s the difference between a synchronized block and a reentrant lock? Let’s understand -

Why Does Unmounting Happen in ReentrantLock

The major difference between a synchronized block and reentrant lock is that the in case of synchronized block the control is handed over to the OS by JVM using syscalls and JVM doesn’t have the complete control over that (also called the intrinsic lock), while in case of reentrant lock the lock is managed by the JVM completely.

So when a virtual thread is waiting for reentrant lock or blocked inside a reentrant lock the JVM allows it to unmount as JVM has all the control, while in case of synchronized block the lock is managed by the OS and since JVM doesn’t have the control over it it doesn’t allow the unmounting and the platform thread is stuck.

To better understand this, we can visualize the difference through animations. The black arrows represent the carrier threads, and the green ones are the virtual threads. This animation is for 2 CPUs with sequential calls to the application as discussed earlier in the blog

Thread Flow With Synchronized block

Here, we can see that both platform threads — one inside the synchronized block (pinned) and the other waiting for the lock (blocked) — are fully occupied. As a result, the system stalls, and no new requests are accepted since only two platform threads are available. This occurs because a synchronized block acquires a lock at the OS level, where the OS manages the lock instead of the JVM, leaving the JVM unable to intervene.

Thread Flow With Reentrant Lock

Here we can observe that when the virtual thread is blocked, the platform thread is unmounted and the same platform thread is reused for different requests. This happens because the reentrant lock is managed by the JVM, and the JVM has all the control over it

Key Takeaways

• Minimize the use of synchronized blocks when working with virtual threads. If synchronization is necessary, ensure that no I/O or blocking operations are performed inside the block.
• Prefer ReentrantLock over synchronized for better scalability and performance, as it allows the JVM to manage locks efficiently without causing thread pinning.
• Handle library-induced synchronization carefully. If a synchronized block comes from a library, such as Caffeine’s LoadingCache in our case, opt for its async variant to avoid blocking. If no async variant is available, consider switching to a library that is either lock-free or supports asynchronous execution.
• Here are some common library alternatives to avoid thread pinning and achieve better scalability:
• Switch from JDBC to R2DBC, which is completely non-blocking.
• If you are using Apache HttpClient or OkHttp, they can cause thread pinning; instead, switch to java.net.http.HttpClient, which is non-blocking internally.
• Use AsyncLoadingCache instead of LoadingCache, as it provides non-blocking behavior.

Conclusion

Virtual threads have revolutionized Java by enabling massive scalability, allowing thousands to millions of lightweight threads to run efficiently while reducing the overhead of traditional OS threads. We’ve successfully optimized multiple production applications using Java 21, but an unexpected load pattern combined with a simple synchronized block led to significant performance degradation.

This challenge provided invaluable insights into debugging and fine-tuning our application for virtual threads. Fortunately, Java 24 has addressed this issue for the synchronized block, making virtual threads even more robust and reliable for high-performance applications.

About the Authors

Saurabh Singh is a senior backend engineer at Games24x7 with over 6 years of experience in software development. He specializes in building and optimizing high-scale distributed applications, focusing on enhancing performance while reducing costs.

Jatin Sinha is working as SDE-2 at Games24x7 and has around 4.5 years of experience. He likes problem-solving and enjoys building and maintaining distributed systems at scale.

Resources -

1. Virtual Threads — https://openjdk.org/jeps/425
2. Synchronize Virtual Threads without Pinning — https://openjdk.org/jeps/491

Production outages in popular online game platforms like My11Circle and RummyCircle can create a huge dent in revenues and cause reputation losses, especially in peak season. We, at Games24x7, aim to ensure at least 99.98% availability of our critical systems. This means that we cannot have an accumulated downtime of more than 1 hour 45 minutes across all our products within the year. While this is achieved by building resilient, fault-tolerant, and highly available systems, at the same time there is a huge role in setting up robust processes that can help in the prevention of production issues, reduction of time to detect and fix issues, and making sure we learn from the past outages. This blog will try to give you insights into the many engineering and operational practices followed at Games24x7 to keep the outages at bay.

Prevention:

1. Having the right attitude: Too much has been written online on best practices to prevent production issues, but mainly it is the mindset that has a big role to play here. At Games24x7, we try to inculcate a sense of ownership in the mind of every engineer. In different forums, leaders often talk about how exponentially we have grown over the years and are still growing. With such a large user base, even one minute of downtime can be extremely detrimental to the business.
2. Effective engineering methods: Our engineers are trained to always optimize for scale and solve for resiliency in every piece of code they write or test. The developers make sure critical flows are not impacted by auxiliary functionalities by appropriate error handling or having separate thread pools. We also have infra-level segregation of different sets of components to keep things isolated and independent. We encourage having on-the-fly configurable switches like feature/functionality level toggles, bypassing slave DBs, etc. This coupled with some other good practices like stringent code reviews, test case walkthroughs, cross-team architecture review forums, minimum 80% unit test coverage, Functional Testing Automation, Performance Tests, User Acceptance Tests, Regression Tests, Canary Deployments, Deployment plan approvals, etc helps us avoid most of the functional and non-functional issues going to production.
3. Planning for major events: To take it a step further in terms of better preparation before major events, we do dry runs on production. This blog by one of my colleagues can help you deep dive into this philosophy of dry runs. In our fantasy platform My11Circle where we see sudden traffic spikes, we don’t rely on the autoscaling of ASGs or HPA alone. Instead, we have built our own Data Science powered scaling system, which can predict the traffic based on the type of the event and current liquidity and pre-scale our systems accordingly.

Monitoring and alerting:

Having an exhaustive monitoring and alerting system in place is extremely crucial. In Games24x7, we monitor different dimensions of our systems:

1. Infrastructure metrics: This mainly covers the usage of resources like CPU, Memory, Disk space, process crashes, etc. Tools used: AWS Cloudwatch, Prometheus, Grafana.
2. Application metrics: This covers the throughput and latencies of different APIs and event consumers of our microservices. We also monitor the different types of error codes emitted from our load balancers and target groups. Tools used: AWS Cloudwatch, Prometheus, StatsD, Graphite, Grafana.
3. Exceptions/Errors: This covers the exceptions or errors occurring in your application code. Tools used: Overops, Harness, Glitchtip, Sentry.
4. Critical Business Metrics: We also plot our critical business metrics like user logins, user registrations, game joins, etc from our microservices. We also detect abnormal trends in traffic patterns using tools like Anodot and our in-house Dynamic monitoring tool.
5. Centralized logs dashboard: We use Last9 which ingests logs from different sources and enables our engineers to search for patterns in the logs for better and quicker debugging.

Most of the above metrics also have corresponding alerts in place and these alerts are integrated with Pagerduty for critical alerts, and with Slack/Email for warnings.

Nipping the issue in the bud:

It is extremely crucial to not overlook any symptoms and give every alert its due, so that a small issue doesn’t snowball into an outage. We have classified the production issues into different categories and defined different Standard Operating Procedures for tackling each of them:

1. Email/Slack alerts: These are the non-critical types of alerts that are not urgent but can be signs of an upcoming major issue or can hamper the experience for some users if not resolved in time. The on-call engineer is supposed to evaluate the alert, do the impact analysis, and create a Jira ticket with the right severity. The severity can help in prioritizing the ticket appropriately.
2. PagerDuty Alerts: These are the alerts that mostly indicate something is wrong with production and needs to be addressed before it gets converted into an outage. These alerts again go to the On-call engineer who has to create a Jira ticket of a higher priority and start the investigation and mitigation. There are run-books in place for common known issues which have happened in the past and are handy to the engineer in our confluence workspace.
3. Production Issues raised by users: These are the Jira tickets that are either raised by our customer support team on the complaints of our end-users or by our business teams. These issues indicate that a set of users are facing a bad experience on the app. The on-call engineer gathers various inputs from the customer support/business teams to gauge the blast radius of the issue, and the functionality that is impacted and accordingly assigns a severity to the Jira ticket. There is an SLA defined for every severity for mitigation and permanent fix.
4. Outages: These are the issues that impact a large number of end users and/or impact the revenue numbers adversely. A Jira ticket is raised on every outage, and with the help of Jira integration with PagerDuty, an alert is generated for all the tech leaders. A Slack channel with a Zoom link and the Jira ticket details is automatically created. The entire team jumps onto the Zoom call to address the outage without any delay. The focus is more on mitigation rather than on permanent fixes in such scenarios. Once the issue is mitigated, then the next natural step is a detailed RCA and a plan for a permanent fix.

Alert Dashboards and Reviews:

As you might have observed in the previous section, every production issue needs to have a corresponding Jira ticket for tracking. Each Jira ticket has an owner and an ETA as per the severity. These Jira tickets are grouped into different Jira Projects and dashboards are created out of those projects. The tech leadership does a weekly review of all these dashboards to make sure that things are moving as per the SLAs and that any delays/blockers are called out. These reviews help us to keep ourselves accountable. The review meeting is also a forum to discuss the Correction of Errors (COE) which I will cover in the next section.

Learning from the mistakes:

Mistakes are part and parcel of our lives, but learning from them and not repeating them is what makes us better from our past selves. Every outage or rollback that happens on production needs to have a COE (Correction of Errors) document. This document has all the details of the outage, the Root Cause Analysis, the learnings and the action items (again tracked as Jira tickets) so that we improve our systems and don’t repeat the same mistakes. The COE document is supposed to be created as soon as the outage is mitigated. The COE document is reviewed with the tech leaders and architects to ensure it is comprehensive. We follow a standard template for the COE doc so that nothing is left to be answered. The template consists of the following sections:

1. Summary
2. Customer/Revenue Impact
3. Detailed Timeline of the issue and the mitigation
4. Questions around the issue discovery:
   - How was the issue discovered?
   - How long did it take to know about the issue from the first impact?
   - Was there a known bug that caused the issue?
   - Were there existing backlog items for this issue?
   - Was this a known failure mode?
   - Was the issue caused by deployment or build release?
   - Was it caused by manual deployment or a configuration change?
   - As a thought experiment what could be done to reduce the time to detection by half?
5. Learnings
6. Corrective actions

Closing Note:

In conclusion, building a resilient team capable of preventing and efficiently responding to outages is not just a technical necessity; it’s also a cultural commitment. At Games24x7, we recognize that our success hinges on our ability to maintain an exceptional user experience, especially during critical times. By fostering a mindset of ownership and accountability among our engineers, implementing robust monitoring systems, and learning from every incident, we continuously improve our processes. With a focus on prevention and proactive engagement, we aim to uphold our promise of 99.98% system availability, ensuring that our platforms remain reliable and enjoyable for all users. Together, we can turn challenges into opportunities for growth, solidifying our reputation as leaders in the online gaming industry.

Author Info

Dhiraj Prajapati is a Senior Engineering Manager at Games24x7, currently building their fantasy sports platform My11Circle. He has been working in the software engineering space for the last 13 years, mainly focussing on distributed systems running at a high scale.

All About Indian Rummy

Rummy is a card game played with two decks of cards, including the two Jokers. To win the game, a player must make a valid declaration by drawing and discarding cards from two piles: a closed deck, where the player cannot see the card they are picking, and an open deck, which consists of cards discarded by players. To win, players need to arrange their cards into valid sequences and sets.

We will discuss in detail on how to maximize wins and minimize losses in the game of Rummy.

Example of a valid declaration of a Rummy Game on rummycircle.com

Terminologies

• Card: A playing card with a rank (1–13) where 11–13 are face cards (J, K, Q) and a suit (♣,◆, ♥, ♠).
• Printed Joker Card: Two special cards apart from 52 cards of the deck that can substitute for any card in a player’s hand.
• Wild Card: A special card drawn at the start of the game that acts similar to the printed joker and can substitute for any card in a player’s hand.
• Pure Sequence (PS): A set of 3 and more cards with consecutive ranks and the same suit. For example, (4♥, 5♥, 6♥, 7♥, 8♥) is a Pure Sequence of length 5, featuring consecutive ranks (4, 5, 6, 7, 8) and the suit ♥.
• Impure Sequence/Sequence (IS or Seq): A set of 3 or more cards with consecutive ranks and the same suit, including one or more Joker cards. For instance, (J♥, 5♣, K♥, A♥) is an Impure Sequence of length 4, with 5♣ serving as the Joker card.
• Set: A combination of 3 or 4 cards with the same rank but different suits. For example, (J♣, J◆, J♥, J♠) is a Set of 4 cards, all with the rank 11 but with four different suits.
• Invalid: Any card combination that does not fit into any of the categories above (PS, IS, Set) is considered Invalid.
• Meld: Any combination of cards that is either a PS, IS, Set, or Invalid.
• Valid Declaration: A grouping with no Invalid melds and at least 2 Sequences, with at least one being a PS. The remaining melds can be any type (PS, IS, Set).
• Open Deck: A stack of face-up cards placed on the table from which players can draw during their turns. This pile is visible to all players, enabling them to make strategic decisions based on the cards shown. It is also referred to as the Discard Pile..
• Closed Deck: A stack of face-down cards remaining after dealing. The cards are arranged randomly and their identities are hidden from the players. Players draw from this deck without knowing which cards they will receive.

How to Play Rummy

Creating valid sequences and sets can be challenging, as the number of combinations increases exponentially with each card dealt to the player, having an upper bound of 13! combinations.

The Point System in Rummy can be found at Rummy Circle.

Tips and Tricks

1. Once the cards are dealt, start grouping them to create both pure and impure sequences; otherwise, it will be more challenging to minimize your points as the game progresses.
2. Focus on hands that include at least one joker along with a pure sequence, as this combination often gives you an advantage early in the game, making it easier to reduce your points.
3. Keep a close eye on what your opponents are selecting from the open deck, as it can offer insights into their hands and what they’re aiming to form.
4. At the start of the game, evaluate the potential for forming pure and impure sequences with your cards. The more cards you have that can help create these sequences, the better your chances of minimizing points.
5. Based on your initial grouping, decide whether to drop the hand. Use the strategies discussed to guide your choice on whether to continue playing.

Therefore, it’s crucial to optimally group your cards with each move to minimize points, which increases your chances of winning and reduces potential losses if you do end up losing.

More tips and tricks here — https://www.rummycircle.com/how-to-play-rummy/rummy-tips-tricks.html

Algorithmic approach to Optimal Grouping

Since our goal is to combine cards in a way that minimizes points, this becomes a combinatorial problem. It can be effectively solved using Dynamic Programming with Bitmask.

Brute Force Approach

The brute force solution involves checking all permutations and grouping the cards into lengths of 3 to 5 for pure and impure sequences, and lengths of 3 to 4 for sets. To illustrate, let’s simplify this with an example using 7 cards instead of 13.

Given hand: 6♣, 7♣, 8♣, 7♠, 9♠, 10♠, 8♠.

There are 7! permutations possible. For example:

• (6♣, 7♣, 8♣, 7♠, 9♠, 10♠, 8♠)
• (6♣, 7♣, 7♠, 8♣, 9♠, 8♠, 10♠)
• (6♣, 7♣, 8♣, 7♠, 8♠, 9♠, 10♠), and so on.

Possible groupings length could be (3, 3, 1), (3, 4) and (5,2). We explain these lengths below.

Let’s take the last permutation (6♣, 7♣, 8♣, 7♠, 8♠, 9♠, 10♠) and consider the grouping (3, 3, 1):

• Meld 1: First 3 cards (6♣, 7♣, 8♣) → Points = 0 (pure sequence).
• Meld 2: Next 3 cards (7♠, 8♠, 9♠) → Points = 0 (pure sequence).
• Meld 3: Last card (10♠) → Points = 10 (invalid).

Total points = 0 (Group 1) + 0 (Group 2) + 10 (Group 3) = 10 points.

Now, using the same permutation (6♣, 7♣, 8♣, 7♠, 8♠, 9♠, 10♠) with the grouping (3, 4):

• Meld 1: First 3 cards (6♣, 7♣, 8♣) → Points = 0 (pure sequence).
• Meld 2: Next 4 cards (7♠, 8♠, 9♠, 10♠) → Points = 0 (pure sequence).

Total points = 0 (Group 1) + 0 (Group 2) = 0, making this an optimal grouping.

Permutations to check 13! * 8 = 49816166400, taking approximately 1.5 hours. Don’t worry we discuss a solution that runs under 10 milliseconds.

Dynamic Programming with Bitmask

The brute force approach is impractical for real-time assistance to players due to its complexity. To improve upon this, we can leverage dynamic programming with bit-masking, which helps manage overlapping subproblems and enables efficient memorization.

Let’s go through the logic along with python code. We provide the helper functions below.

Imports:

import numpy as np
import copy
import random
import itertools
from scipy.ndimage import shift

Hand points calculation logic:

# locations - store the locations of the card in the hand
# hand_array - store the hand as a tuple of suit and card
# hand - store the hand as array of string
# joker - joker card
def calculcate_hs(locations,hand_array,hand,joker):
    sc = 0
    for loc in locations:
        if not (hand_array[loc][0] == -1 or hand_array[loc][1] == int(joker[:-1])):
            if hand_array[loc][1] == 1:
                sc+=10
            else:
                sc+=min(10,hand_array[loc][1])
    return sc

Get count of cards already processed:

# get the number of the cards that has been considered in the hand
def get_len(num):
    s=0
    while num!=0:
        s+=(num & 1)
        num = num>>1
    return s

Get the location of the cards already considered:

# get all the cards position that are considered in the hand
def get_set_pos(num):
    i = 0
    pos = []
    while num!=0:
        if (num & 1) == 1:
            pos.append(i)
        num = num>>1
        i+=1
    return pos

Make a suit to cards and cards to suit mapping present in the hand:

# get suit to card mapping dictionary both including and excluding the wild card
def get_suit_card_mapping(hand0,joker):
    different_suits = {'h':[], 's':[], 'c':[], 'd':[]}
    different_suits_without_joker = {'h':[], 's':[], 'c':[], 'd':[]}
    for i,card in enumerate(hand0):
        initial = card[0]
        remaining = int(card[1:])
        different_suits[initial].append(remaining)
        if card in joker or card == 'j':
            continue
        else:
            different_suits_without_joker[initial].append(remaining)
    for card_type in different_suits:
        different_suits[card_type] = sorted(different_suits[card_type])
        different_suits_without_joker[card_type] = sorted(different_suits_without_joker[card_type])
    return different_suits,different_suits_without_joker
# get card to suit mapping dictionary
def get_card_suit_mapping(hand0,joker):
    different_suits = {1:[], 2:[], 3:[], 4:[], 5:[], 6:[], 7:[], 8:[], 9:[], 10:[], 11:[], 12:[], 13:[]}
    for i,card in enumerate(hand0):
        initial = card[0]
        remaining = int(card[1:])
        if card in joker or card == 'j':
            continue
        else:
            different_suits[remaining].append(initial)
    return different_suits

Store all the hash in relevant meld type — Pure Sequence, Impure Sequence and Sets. ‘typ’ 1 is pure sequence, ‘typ’ 2 is impure sequence and ‘typ’ 3 are sets in the code below:

# add cards which are considered in the hand based on type pure seq/impure seq/set in the good hands which will be part of the optimal grouping
def mark_location(comb,loc_mapping,idx,mask,good_hands,typ):
    if len(comb) == idx:
        good_hands[typ].add(mask)
    else:
        if type(comb[idx]) == tuple:
            pos = 2**comb[idx][1]
            mark_location(comb,loc_mapping,idx+1,mask+pos,good_hands,typ)
        else:
            for i in loc_mapping[comb[idx]]:
                pos = 2**i
                mark_location(comb,loc_mapping,idx+1,mask+pos,good_hands,typ)

Lets Begin,

First, let’s explore the combinatorial aspect of this problem. Since each player is dealt 13 cards, we can represent each card with a unique bit which behaves as a hash for the card, resulting in ²¹³ = 8192 possible combinations for card selection. For example, if a player has the hand (5♣, 6♣, 7♣, 8♣, 9♣, 7♠, 8♠, 9♠, 10♠, J♥, K♥, A♥, K♦), we can assign bits as follows:

• 5♣ is represented by the 0th bit and can be represented as ²⁰ = 1,
• 6♣ by the 1st bit and can be represented as ²¹ = 2,
• 7♣ by the 2nd bit and can be represented as ²² = 4,
• 8♣ by the 3rd bit and can be represented as ²³ = 8,
• 9♣ by the 4th bit and can be represented as ²⁴ = 16

and so on for the remaining cards.

We have established a method to uniquely represent and identify each card in the hand. Now, let’s discuss how to represent melds. For instance, consider the first three cards of the hand mentioned earlier — (5♣, 6♣, 7♣). We can create a unique hash for this meld by adding the unique representations of each card: (1 + 2 + 4) = 7. This sum represents the unique hash for the meld (5♣, 6♣, 7♣).

Now, let’s see how we store the melds using these hashes. Consider the hand (5♣, 6♣, 7♣, 8♣, 9♣, 7♠, 8♠, 9♠, 10♠, J♥, K♥, A♥, K♦):

1. Hash for the pure sequence (5♣, 6♣, 7♣, 8♣, 9♣):
   (1 + 2 + 4 + 8 + 16) = 31
2. Hash for the pure sequence (7♠, 8♠, 9♠, 10♠):
   (32 + 64 + 128 + 256) = 480
3. Hash for the pure sequence (J♥, K♥, A♥):
   (512 + 1024 + 2048) = 3584
4. Hash for the last invalid meld (K♦):
   (4096) = 4096

To find the total hash for the hand, we sum these values:
31 + 480 + 3584 + 4096 = 8191.
As you can see that grouping cards and calculating the hash will always sum to 8191. We can conclude that if we permute through all the combinations of meld in the hand and sum up their hash, we will get 8191 giving us a way to memorize all possible pure sequences, impure sequences and sets.

Steps of the algorithm

To determine the optimal grouping and points for the hand, we can follow these steps:

Find All Possible Pure Sequences:

• Generate pure sequences of lengths 3 to 5 and store their hashes.
• If no pure sequences are found, calculate the total points for the hand.

Check for Impure Sequences:

• If at least one pure sequence exists, find all possible impure sequences and store their hashes.
• If no impure sequences are found, check for non-overlapping pure sequences calculated in Step 1. If none are found, calculate the points for the cards not included in the only pure sequence.

Evaluate Non-Overlapping Pure Sequences:

• If there are at least two non-overlapping pure sequences or one pure and one impure sequence, identify all possible sets in the hand and store their hashes.

Dynamic Programming Approach:

• Once all sets and sequences are calculated, utilize dynamic programming to explore all valid combinations of these sets and sequences, optimizing for the lowest total points.

By systematically approaching the problem in this manner, we ensure that we cover all potential combinations and can determine the best grouping of cards effectively.

Lets Code the algorithm,

Let’s illustrate the pre-processing of a hand by converting it into mappings of suit to card and card to suit.

# hand of the user
hand = ['1h', '2h', '3h', '4h', '1h', '2h', '3h', '4h', '4d', '12h', '13h', '5h', 'j']
# wild card
joker = '4d'
if joker == 'j':
    joker = '1h'
# pre-processing of the hand and joker
hand_inverted = [card[-1]+card[:-1] if len(card)>1 else 'j' for card in hand]
hand_array = [[card[-1],int(card[:-1])] if len(card)>1 else [-1,20] for card in hand]
joker_inverted = ['s'+joker[:-1],'c'+joker[:-1],'d'+joker[:-1],'h'+joker[:-1]]
hand_temp = []
all_jokers = []
loc_mapping = {}
for i,h in enumerate(hand_inverted):
    if h in joker_inverted or h == 'j':
        all_jokers.append((h,i))
    if h in loc_mapping:
        loc_mapping[h].append(i)
    else:
        if h != 'j':
            hand_temp.append(h)
        loc_mapping[h] = [i]
hand_inverted = hand_temp
different_suits,different_suits_without_joker = get_suit_card_mapping(hand_inverted,joker_inverted)
different_cards = get_card_suit_mapping(hand_inverted,joker_inverted)
good_hands = {1:set(),2:set(),3:set()}
# calculate initial hand score
output = [calculcate_hs(range(len(hand)),hand_array,hand,joker),[2**i for i in range(0,len(hand))]]
final_output = []

Find All Possible Pure Sequences, we get pure sequences and convert it to a hash and store it into a map.

# logic to get all possible pure sequences in the hand
def get_pure_sequences(different_suits):
    all_possible_pure_seqeunces = []
    for length in range(3,6):
        for card_type,cards in different_suits.items():
            current_seq = []
            if 1 in cards:
                cards = cards + [14]
            for card in cards:
                if len(current_seq) == 0 or current_seq[-1]+1 == card:
                    current_seq.append(card)
                else:
                    current_seq = [card]
                if len(current_seq)==length:
                    all_possible_pure_seqeunces.append([f"{card_type}{1 if i == 14 else i}" for i in current_seq])
                    current_seq = current_seq[1:]
    return all_possible_pure_seqeunces
# we first look for at least one pure sequences
seq = get_pure_sequences(different_suits)
# we store all the pure sequences in a map
for i in seq:
    mark_location(i,loc_mapping,0,0,good_hands,1)

Implementing Check for Impure Sequences and Evaluate Non-Overlapping Pure Sequences, we get impure sequences and if impure sequences exists we also calculate sets. We convert it to a hash and store it into a map as we did for the pure sequences above.

Helper function to check whether a sequence in impure or not and to get all possible sets.

# function to check where a sequences is an impure sequence or not.
def check_impure_seq(l,jokers):
        if len(l)>1:
            d = shift(l,1,cval=l[0]-1)+1
        else:
            d = l
        s = sum(l-d)
        if s<=len(jokers):
            return s
        elif 1 in l:
            l[l==1] = 14
            l = np.sort(l)
            if len(l)>1:
                d = shift(l,1,cval=l[0]-1)+1
            else:
                d = l
            s = sum(l-d)
            if s<=len(jokers):
                return s
        return -1
# function to get all possible sets in the hand
def get_sets(different_suits,all_jokers,joker_combinations):
        all_possible_sets = []
        # below we generate combinations for cards with same number but different suits
        for card in different_suits:
            if len(different_suits[card]) >= 3:
                for length in range(3,len(different_suits[card])+1):
                    combs = itertools.combinations(different_suits[card],length)
                    for comb in combs:
                        s = []
                        for val in comb:
                            s.append(str(val)+str(card))
                        all_possible_sets.append(s)
        # below we generate combinations for cards with same number but different suits and also including the joker
        for length in range(max(2,3-len(all_jokers)),4):
            for joker_length in range(max(1,3-length),min(4-length+1,len(all_jokers)+1)):
                for card,card_type in different_suits.items():
                    if len(card_type)>=length:
                        combs = list(itertools.combinations(card_type,length))
                        for comb in combs:
                            current_seq = []
                            for val in comb:
                                current_seq.append(str(val)+str(card))
                            for item in joker_combinations[joker_length-1]:
                                all_possible_sets.append(current_seq+list(item))
        return all_possible_sets

We iterate through all the combinations involving joker and check if it is an impure sequence or not.

# now we create optimal hands based on valid declaration.
# if we have at least one pure sequence then only we can make a valid declaration.
if len(seq)>0:
    set_flag = False
    # now we check if there is at least one impure sequence or non-overlapping pure sequence.
    joker_combinations = [list(itertools.combinations(all_jokers,i)) for i in range(1,min(6,len(all_jokers)+1))]
    if len(all_jokers)>0:
        seq = []
        for t in range(2,len(joker_combinations)):
            for p in joker_combinations[t]:
                seq.append(list(p))
        for card_suit,cards in different_suits_without_joker.items():
            for x in range(max(1,3-len(all_jokers)),5):
                vals = itertools.combinations(cards,x)
                for val in vals:
                    val = list(val)
                    t = check_impure_seq(np.array(val),all_jokers)
                    tt = [f"{card_suit}{itr}" for itr in val]
                    if t != -1:
                        for i in range(max(1,t)+max(0,(3-x)-1),min(5-x+1,len(all_jokers)+1)):
                            for p in joker_combinations[i-1]:
                                seq.append(tt+list(p))
        for i in seq:
            mark_location(i,loc_mapping,0,0,good_hands,2)
        good_hands[2] = good_hands[2]-good_hands[1]
        # if there is an impure sequences we look for sets in the hand.
        if len(seq)>0:
            set_flag = True
            seq = get_sets(different_cards,all_jokers,joker_combinations)
            for i in seq:
                mark_location(i,loc_mapping,0,0,good_hands,3)
    # if we have at least two pure sequences and sets in the hand.
    if len(seq)>1 and not set_flag:
        seq = get_sets(different_cards,all_jokers,joker_combinations)
        for i in seq:
            mark_location(i,loc_mapping,0,0,good_hands,3)
# code below is part of the if block of the pure sequence which is 
# the first if block of this section.

Continuing in the same if section as above, we perform memorization of each meld and store the best possible valid combination with each grouped meld.

# in the same if block as above.
    ans = [200]*2**len(hand)
    optimal_hand = [None]*2**len(hand)
    # below we look for a set of non-overlapping pure/impure/sets that gives the minimum score.
    def get_permutation(current_hand,optimal_hand,ans,hand_array,hand,joker,good_hands):
        if current_hand == 0:
            return 0, []
        elif ans[current_hand] != 200:
            return ans[current_hand],optimal_hand[current_hand]
        else:
            flag = False
            for tc in [1,2,3]:
                for pos in good_hands[tc]:
                    if pos&current_hand == pos:
                        flag = True
                        nscore,o = get_permutation(current_hand^pos,optimal_hand,ans,hand_array,hand,joker,good_hands)                        
                        if nscore<ans[current_hand]:
                            ans[current_hand] = nscore
                            optimal_hand[current_hand] = [pos] + o
            if not flag:
                pos = get_set_pos(current_hand)
                nscore = calculcate_hs(pos,hand_array,hand,joker)
                ans[current_hand] = nscore
                optimal_hand[current_hand] = [current_hand]
            return ans[current_hand],optimal_hand[current_hand]
    # below we map the hash to the card and calculate the hand score.
    hand_max = 2**len(hand)-1
    for j in good_hands[3]:
        ans[j] = 0
        optimal_hand[j] = [j]
    for i in good_hands[1]:
        ans[i] = 0
        optimal_hand[i] = [i]
        left_hand = hand_max^i
        score = calculcate_hs(get_set_pos(left_hand),hand_array,hand,joker)
        if score<output[0]:
            output[0] = score
            output[1] = [i] + [left_hand]
    for j in good_hands[2]:
        ans[j] = 0
        optimal_hand[j] = [j]
    flag = False
    for i in good_hands[1]:
        for j in good_hands[2]:
            if i&j==0:
                ans[i|j] = 0
                optimal_hand[i|j] = [i,j]
                score,oh = get_permutation(hand_max^(i|j),optimal_hand,ans,hand_array,hand,joker,good_hands)
                if score<output[0]:
                    output[0] = score
                    output[1] = [i,j] + oh
                if score == 0:
                    flag = True
        if flag:
            break
        for j in good_hands[1]:
            if i&j==0:
                ans[i|j] = 0
                optimal_hand[i|j] = [i,j]
                score,oh = get_permutation(hand_max^(i|j),optimal_hand,ans,hand_array,hand,joker,good_hands)
                if score<output[0]:
                    output[0] = score
                    output[1] = [i,j] + oh
                if score == 0:
                    flag = True
        if flag:
            break
    for output_hand in output[1]:
        h = []
        for location in get_set_pos(output_hand):
            h.append(hand[location]) 
        final_output.append(h)
print('Grouped Hand = ' + str(final_output))
print('Hand Points = ' + str(min(80,output[0])))

The above code has a computational complexity of ²¹³ + ²¹³ + ²¹³= 24576 with average case complexity of Θ(²¹³) and runs under 10 milliseconds.

Conclusion

We’ll explore the most popular online RMG of Rummy, sharing tips and tricks to maximize wins and minimize losses. We’ll also address the challenge of optimal grouping using dynamic programming with bit-masking, along with the relevant code.

Author

Naman Lal — He specialises in leveraging machine learning and artificial intelligence to transform data into actionable insights. With a focus on automating decision-making and solving complex problems, he is dedicated to driving innovation and efficiency in the tech landscape.