Stories by Dream11 Engineering on Medium

DRS: Making high-impact & informed decisions with Dream11’s In-house Experimentation Platform!

Dream11 Engineering — Thu, 23 Jan 2025 09:50:55 GMT

With millions of Indian sports fans across our Dream Sports ecosystem, it takes a highly data-driven team to understand what is required for great fan engagement and ultimately making sports better through our various brands. At Dream11 too, we have been on a constant mission to provide amazing end-to-end user experiences to over 130 million sports fans with new product features and offerings that are shipped regularly. Data at the core of every decision that we take as Dreamsters. We run multiple experiments parallelly at Dream11 to evaluate a new feature or product customisation for our users. To enable better decision-making, all these experiments are run by our product and tech team on our very own in-house experimentation platform Data-Driven Review System or as we call it, DRS.

Why did we build DRS?

On the Dream11 app, the spike in the traffic can be mammoth during landmark sporting events like the IPL. With such a huge volume and large scale user activation events, the complexity of experimentation multifold. At any given time, teams at Dream11 are running numerous parallel experiments, testing out innumerable hypotheses. Thus, there arises a need to conduct these product increment tests on an isolated set of similar users so multiple experiments don’t bias each others’ results and accurately attribute the change in a data metric for every product increment.

To figure out a resolution, we played with multiple external tools for experimentation but weren’t able to find one that managed and aligned with our needs and scale. Internal teams had various custom use-cases around user sampling, assignment strategies and backend experimentation capabilities that were missing in the market, which pushed us to build our own internal platform.

The building blocks of a DRS experiment

Type of Experiments

A DRS experiment can be one of the following types:

Frontend
Backend

A frontend experiment is essentially on the user interface. In simpler terms, frontend experiments are designed to experiment on cosmetic changes to your application (for example, the buttons on the app or exposure to a new feature section).

Backend experiments, on the other hand, are experiments which modify the data the user sees. It usually involves different input parameters to algorithms and business logic, driven by Application Programming Interface (API) responses.

Impact Area: Cohorts

A controlled user space is imperative in reducing the possibility of an adverse impact of an experiment on any critical business metric. This essentially means that it is necessary to restrict the impact area of the experiment.

DRS provides out-of-the-box support for internal teams through a robust and customisable Cohorting Engine. In addition to that, DRS provides support for complex sampling strategies based on user traits (which we will cover in detail below). Stakeholders can also create a static list of users to run experiments on by running queries on historic data.

In addition to the on-the-fly cohort creation, there are some user sets that are commonly used for experimentation within Dream11. The DRS platform also provides these system cohorts to stakeholders with ease. Some of them are:

All: All Dream11 Users
Logged In: Only Authenticated Users
Non-Logged In: Only Non Logged In Users
Internal Users: All Dream11 Employees

Further, users have the power to put more advanced configurations on the cohort with the help of the Include/Exclude functionality.

User Variant Assignment

Most of the use-cases, whether frontend or backend, require a real-time assignment of the user to an experiment. Through this, a user becomes a part of an experiment cohort only if they open the application.

Having said that, DRS provides support for the pre-assignment of users to an experiment for use-cases where batch processing is involved. This is for experiments like testing the validity of machine learning models, personalisation and recommendation algorithms, pre-computed promotions strategies, and mast-blast communications.

External Integrations

One of the major charters for us while building DRS was to make it in a highly modular manner so that various blocks can have a plug-and-play functionality of their own. Hence, DRS obliges if one customer wants to use their own user-experiment assignment logic.

Thus, a wide variety of sampling scenarios are possible through DRS.

Sampling Techniques

Cluster

User population is first divided into groupings or clusters via some clustering algorithm or ML model. Then, a cluster is selected in its entirety based on assignment strategy and attributed to the variant as per the specified allocation. At the end, all users belonging to a cluster will only get mapped to one of the variants.

Manual Mapping

There are experiments where mapping of these clusters has to be precise. Then, clusters are manually mapped to variants as defined by the user.

Stratified

User population is first split into various mutually exclusive, homogeneous and non-overlapping strata. Users with similar affinity define a stratum. Let’s say you want to use 10 user properties and 5 computed attributes of the user for identifying strata. Each unique combination of the above 15 traits will become a stratum. Then the sample is drawn according to an assignment strategy from each stratum to be attributed to variants as per defined ratios.

Assignment Strategies

Randomised

A user gets assigned one of the experiment variants selected in a random fashion weighted by the allocation percentages of variants. Even though this is a reliable strategy for most assignments, in case of cluster sampling, this can produce skewed assignments due to non-uniformity in cluster sizes.

Round Robin

The variants allotted to a user are rotated for each request based on their percentages. We maintain an overshoot map that denotes how much the current allocation of a variant exceeds the intended allocation. We try to reduce this by allotting the variant that has least overshoot value. This value is stored in a local cache, in order to handle the large amount of requests. And this value is periodically synced with a database to maintain balance across the application servers

Isolation amongst experiments

Experiments produce data! The concern though is if we can have confidence in the data produced through the experiment. Is it reliable?

For the data to be reliable, it has to be guarded and kept isolated from external factors that can have a bearing on the data metrics. The isolation should be based on two key factors:

Users

We need to maintain exclusivity of the users being marked to a particular experiment. In simpler terms, the users assigned to one experiment should not be assigned to any other experiment.

Slow-release with traffic exposure

With hundreds of experiments running in parallel, it’s paramount to have a non-buggy user experience. DRS allows us to control the exposure of the traffic so as to spread out the impact of the experiment over a period of time. In case of any issues with the experiment, the impact is minimised. We can use our exposure field to control user traffic that will be exposed to the given experiment. On cautiously assessing things, we gradually increase exposure to 100%.

Our source of inspiration for this was the canary deployment process where user traffic is gradually moved to a newly deployed stack.

Ensuring Threshold

Threshold is configured for an experiment to maintain a check on the maximum number of users it can be exposed to. As there can be millions of users in a cohort, if we want to expose this experiment to a limited number of users, we can use threshold. This is especially crucial in exclusive experiments as they hoard up an unnecessarily large number of users. As millions of requests per minute are served, simply updating the count would create a hotkey problem. Given the business, confirm if the threshold is used to ensure users are not over-allocated to few experiments.

Tech@Dream11

Challenges

The traffic on Dream11 during big-ticket events can go from thousands of concurrent users to millions in just a few minutes and our edge services handle over 120 million requests per minute (RPM) during peak load.

We had the challenge of designing a highly tunable experimentation system that can scale at ultra-low latency. Before serving experiments we had to address these key problems:

Segregating relevant requests, such that the experimentation service does not deal with such high RPM
Experiments are relatively long-running and user to variant stickiness is maintained throughout an experiment’s lifetime

Implementation Details

As specified earlier, experiments are served to users in two modes:

Frontend
Backend

Experiments are configured on unique paths. These unique paths are the page names for the frontend and API paths for the backend.

The frontend experiments are driven by API requests from clients on page load/app launch. For frontend assignments, multiple paths are requested in one request at app launch.

For backend experiments, the experiment information is injected in the API request header.

DRS responds back with the variant information each user is assigned. It’s now a responsibility of the frontend client or the backend microservice to interpret those config variables based on their unique business logic.

Optimisations

Managing Traffic

We introduced tokens and added a heuristic at the API gateway layer using an Experiment SDK. The token contains experiment mapping and is circulated with each request which is inspected at API gateway and forwarded to experiment service only if users are eligible for a new assignment.

Managing Data

The experiment service uses Cassandra (a free and open-source, distributed, wide-column store) as the source of truth and Aerospike (a flash memory and in-memory open source) as its cache. We ensure that the system scales linearly with the ever-growing experimentation adaptation, and the data is synced with Amazon Redshift (a data warehouse) to be used for all analytical processes.

All data is denormalized to have pointed queries. Some data that is updated infrequently is also kept in the local cache and is periodically updated in background. This also guards us against the thundering herd problem.

Performance Numbers

We receive bursts in our traffic just before a match starts, with this effect getting exaggerated during the IPL. Our experiment service has been able to serve a peak load of 16M requests with an API latency of p95 < 10ms.

The below graph shows the spiky request patterns that the experimentation service handles with best-in-class performance:

Future of DRS

At Dream11, we have just begun our journey in the world of experimentation with DRS and there is a long way to go. Here are some significant milestones that we have envisioned for the immediate future:

Augmented Analytics

Tracking of primary and secondary metrics for experiments, totally integrated with DRS and real-time!

Bandit Testing

Intelligently auto-scaling the attribution of variants to a cohort to finally make the best performing variant the default version.

No-code Experimentation

A low-code, no-code capability for non-technical stakeholders to create UI-driven experiments.

Remote Config

Providing visibility and fine-grained control over app’s behaviour and appearance so one can make changes by simply updating configurations in DRS, and easily turn feature exposure on and off.

Are you interested in solving problems related to data at multi-petabytes’ scale, handling and processing billions of data points per day or working with passionate and innovative minds on the turf? We are currently hiring across all levels! Apply here to join us. More exciting stuff to come from the Experimentation Team at Dream11. Stay tuned!

DRS: Making high-impact & informed decisions with Dream11’s In-house Experimentation Platform! was originally published in Dream11 Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

DRS - Making high-impact & informed decisions with Dream11’s In-house Experimentation Platform!

Dream11 Engineering — Thu, 05 May 2022 05:15:48 GMT

Data at the core of every decision that we take as Dreamsters. We run multiple experiments parallelly at Dream11 to evaluate a new feature or product customisation for our users. To enable better decision-making, all these experiments are run by our product and tech team on our very own in-house experimentation platform Data-Driven Review System or as we call it, DRS.

Why did we build DRS?

On the Dream11 app, the spike in the traffic can be mammoth during landmark sporting events like the IPL. With such a huge volume and large scale user activation events, the complexity of experimentation multi-folds. At any given time, teams at Dream11 are running numerous parallel experiments, testing out innumerable hypotheses. Thus, there arises a need to conduct these product increment tests on an isolated set of similar users so multiple experiments don’t bias each others’ results and accurately attribute the change in a data metric for every product increment.

How does the DRS work at Dream11

The building blocks of a DRS experiment

Type of Experiments

A DRS experiment can be one of the following types:

Frontend
Backend

Impact Area: Cohorts

DRS provides out-of-the-box support for internal teams through a robust and customisable Cohorting Engine. In addition to that, DRS provides support for complex sampling strategies based on user traits (which we will cover in detail below). Stakeholders can also create a static list of users to run experiments on by running queries on historic data.

All: All Dream11 Users
Logged In: Only Authenticated Users
Non-Logged In: Only Non Logged In Users
Internal Users: All Dream11 Employees

Further, users have the power to put more advanced configurations on the cohort with the help of the Include/Exclude functionality.

User Variant Assignment

External Integrations

Thus, a wide variety of sampling scenarios are possible through DRS.

Sampling Techniques

Cluster

Manual Mapping

There are experiments where mapping of these clusters has to be precise. Then, clusters are manually mapped to variants as defined by the user.

Stratified

Assignment Strategies

Randomised

Round Robin

Isolation amongst experiments

Experiments produce data! The concern though is if we can have confidence in the data produced through the experiment. Is it reliable?

For the data to be reliable, it has to be guarded and kept isolated from external factors that can have a bearing on the data metrics. The isolation should be based on two key factors:

Users

We need to maintain exclusivity of the users being marked to a particular experiment. In simpler terms, the users assigned to one experiment should not be assigned to any other experiment.

Slow-release with traffic exposure

Our source of inspiration for this was the canary deployment process where user traffic is gradually moved to a newly deployed stack.

Ensuring Threshold

Tech@Dream11

Challenges

We had the challenge of designing a highly tunable experimentation system that can scale at ultra-low latency. Before serving experiments we had to address these key problems:

Segregating relevant requests, such that the experimentation service does not deal with such high RPM
Experiments are relatively long-running and user to variant stickiness is maintained throughout an experiment’s lifetime

Implementation Details

As specified earlier, experiments are served to users in two modes:
Frontend
Backend
Experiments are configured on unique paths. These unique paths are the page names for the frontend and API paths for the backend.
The frontend experiments are driven by API requests from clients on page load/app launch. For frontend assignments, multiple paths are requested in one request at app launch.
For backend experiments, the experiment information is injected in the API request header.
DRS responds back with the variant information each user is assigned. It’s now a responsibility of the frontend client or the backend microservice to interpret those config variables based on their unique business logic.

Optimizations

Experiment SDK

Managing Data

Performance Numbers

We receive bursts in our traffic just before a match starts, with this effect getting exaggerated during the IPL. Our experiment service has been able to serve a peak load of 16M requests with an API latency of p95 < 10ms.

The below graph shows the spiky request patterns that the experimentation service handles with best-in-class performance:

Future of DRS

1. Augmented Analytics

Tracking of primary and secondary metrics for experiments, totally integrated with DRS and real-time!

2. Bandit Testing

Intelligently auto-scaling the attribution of variants to a cohort to finally make the best performing variant the default version.

3. No-code Experimentation

A low-code, no-code capability for non-technical stakeholders to create UI-driven experiments.

4. Remote Config

Providing visibility and fine-grained control over app’s behaviour and appearance so one can make changes by simply updating configurations in DRS, and easily turn feature exposure on and off.

Are you interested in solving problems related to data at multi-petabytes’ scale, handling and processing billions of data points per day or working with passionate and innovative minds on the turf? We are currently hiring across all levels! Apply here to join us. More exciting stuff to come from the Experimentation Team at Dream11. Stay tuned!

Authors:-
Shray Jaiswal, Srijan Gupta , Hrithik Bhardwaj, Tarun Tyagi, Farhan Ali

Optimising price performance at Dream11 with GP3

Dream11 Engineering — Mon, 28 Feb 2022 08:03:33 GMT

~By Siddharth Terse & Parth Ingole

Introduction

Amazon Web Services (AWS) announced their next generation general-purpose Solid State Drives (SSD) volumes in December 2020, aka GP3. In the previous version GP2, high throughput and Input/Output Operations Per Second (IOPS) were required for better performance, and a large storage or IOPS volume was needed to be provisioned. This usually led to the wastage of storage resources at much higher costs.

In GP3, there is no need to scale the additional block storage capacity for managing IOPS. The SSD is independent from storage capacity. Moreover, a higher IOPS and throughput can be added for better performance. When we migrated to GP3 from GP2 at Dream11, we observed an increase in latency performance by upto 18%. We also noticed better price-performance optimisation. Although, this may vary depending on the storage and IOPS needs with different applications, datastore and virtual desktop use cases.

To understand how Dream11 benefitted from this migration, let us first understand the key differences between GP2 and GP3.

A volume-type comparison of GP2 and GP3:

Improvement in performance :

Once the migration process had concluded, we observed substantial improvement in latency and the performance of the Central Processing Unit (CPU) while using GP3.

While testing a key aspect of the Dream11 application while transitioning from GP2 to GP3, we found that at 40 million Requests Per Minute (RPM), the performance of GP3 was better than GP2, even for smaller volumes like 100 GB. The below tables and graphs present a brief overview of the results.

Service A tested at 40 million RPM with GP2 and GP3:

Service A latency at 40 million RPM with GP2

Service A latency at 40 million RPM with GP3

Impact of GP3 on Dream11

As depicted above, with GP3, we were able to achieve up to 92% cost reduction in our Elastic Block Store (EBS) and saw a significant improvement in the overall performance in various use cases. Here are some examples.

Better price-performance optimisation:

Let’s assume that we provisioned a large disk on our custom datastore, just to get better IOPS using GP2. For example, we want a mysql datastore with 100 GB but on 3000 IOPS. Then,

With GP2, we need to provision 1000 GB
With GP3, we need to provision 100 GB with about 92% lower costs

The below table indicates the benefits of GP3 over GP2, for various IOPS requirements.

Increased impact on direct storage:

Let’s assume that we have similar storage requirements for both GP2 and GP3 use cases. GP3 is 20% cheaper for smaller volumes whereas the IOPS requirement is below 3000. At Dream11, we get this benefit across 100 applications with tens of thousands of servers running on GP2. For a few applications like Kafka where we need a large disk as well as high IOPS, GP3 still costs about 8% lower.

Comparative costs between GP2 and GP3 in us-east-1:

At Dream11, we created a separate squad for ‘Price Performance Engineering’ within our SRE team. With the migration to GP3 and a dedicated team tracking price performance optimisation, so far, results have been awesome!

Stay tuned for our other output enhancement initiatives such as Cost Wastage Portal, Adoption of Graviton, Adoption of Granulate and more!

Keen to work and innovate for the world’s largest fantasy sports platform with over 120 million users? Join us at Dream11 by applying here!

Optimising price performance at Dream11 with GP3 was originally published in Dream11 Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Data Feast — A Highly Scalable Feature Store at Dream11

Dream11 Engineering — Mon, 21 Feb 2022 15:00:25 GMT

Data Feast — A Highly Scalable Feature Store at Dream11

At Dream11, we have seen tremendous growth in terms of our user base to become the world’s largest fantasy sports platform. Our users can demonstrate their knowledge and skills in sports and connect deeper with the sport they love. With an ever-growing user base that currently stands at over 120 million, we work meticulously on our data platform to handle the sheer scale of generated data every second. Our goal is to build self-serve data products, be a data-obsessed team, and promote the concept of ‘data as a service within the organisation.

We started building a feature store to cater to our Machine Learning (ML) and in-house customer data platform needs, staying true to our goal. Here’s our journey of building Data Feast — our feature store product built entirely in-house.

What is a feature store?

A feature store is a data platform product that makes it easy to build, search, deploy, and use the features for various use-cases.

Here is a quick refresher on what is a feature, a feature value, and feature group/entities

A feature store is a central framework that processes, stores, and serves data that powers various use-cases of ML models and our customer data platform. Feature store helps transform raw data into feature values, store them, and serve them for model training and online inferencing. They also help reduce feature duplication by providing a registry/catalog for searching feature details. By automating all these steps, the feature store helps minimise duplication of data engineering efforts, speed up the ML models’ time to production, and help improve collaboration across data science teams.

Feature Store components

Five significant components that together form a feature store are:

1. Storage: This is the heart of a feature store. It provides capabilities for storage and retrieval of feature data for training and inference purposes.

2. Serving: The serving layer is responsible for serving feature data to models in a consistent format across training and inferencing.

3. Registry: A centralized registry is one of the essential components of a feature store. It enables standardized feature definitions and metadata for discovery and serves as a single source of truth for information about a feature.

4. Monitoring and Alerting: Usually, when something goes wrong with an ML setup, it mainly points to a data problem. A feature store should detect and raise alerts if such issues arise. Monitoring and alerting calculates all metrics on data stored and served, describing the correctness and quality of the features in question.

5. Transformation: In all ML applications, we require capabilities to process data to generate features that the models can then consume for inferencing and prediction. Feature stores should thus have out-of-the-box capabilities to perform data transformations that produce these features.

The Feature Store at Dream11, aka Data Feast

As we at Dream11 began experimenting, adapting, and creating many intelligent ML models, we saw exponential growth in the features generated by our various teams for multiple use-cases. Every team developed hundreds and thousands of features across multiple domains for various individual projects. We felt a need to build a system that can manage and handle all requirements of storing and serving features in a consolidated, easy, effective, standard way across all teams, projects, and models. We also foresaw the need to build a central repository that helps discover features and their metadata, thus reducing duplicate efforts across teams.

With all this in mind, we set out to build a system that would help solve all the above problems, and this is how Data Feast was envisioned and brought to life at Dream11.

Where we currently stand with Data Feast

Here is how we built the five significant components of feature stores that we talked about earlier.

Storage

In terms of storage requirements, there are two primary storage tiers in a feature store — online and offline to support different needs of inferencing and training of models.

The online storage tier

The online storage tier should have high throughput potential (which usually provides multi-million read requests per second) and low latency (single-digit millisecond latency) read capabilities. These are typically implemented with key-value stores. An online store usually has the latest version of the feature value for a given entity in a feature.

For the first iteration of the online store, we decided to use HBase with Apache Phoenix due to its scalability in terms of storage and high ingestion throughput with low latency reads. The additional feature of running SQL queries using the Phoenix interface on top of the feature data was an advantage for specific use cases that were being developed. We have heavily used and built systems and frameworks with HBase and Phoenix to solve a lot of requirements for high throughput ingestion and quick retrieval on terabytes of data.

As the adoption and the scale of data and usage grew, we noticed some bottlenecks in performance w.r.t throughput, majorly due to the limitation of phoenix query servers on the EMR master nodes, which led to us rethinking our online store strategy.

After much brainstorming, we concluded that a single store would not solve all our requirements perfectly. A store like HBase with Phoenix gave us decent throughput with an added functionality of SQL aggregate querying for specific situations. Still, it did not meet our expectations on latency and throughput after reaching features and data at scale. We decided to move to a hybrid online store model, so we could dynamically introduce an additional online store based on the use case at hand. This way, we can’t be held back by a store and its limitations and use the best parts of multiple stores to solve all kinds of use cases.

During our search for a store that can help solve our very high throughput, single-digit millisecond latency requirement, we came across RonDB. It is the open-source distribution of the NDB Cluster. RonDB claimed to provide LATS: low Latency, high Availability, high Throughput, scalable Storage, and it had good benchmark numbers to back its claim (more details on benchmarks available here). We performed a proof of concept (POC) with RonDB and realized it would work for us based on benchmark numbers. RonDB (NDB — Network Database) is accessible via a MySQL server cluster setup which was again a plus point for us regarding easy scalability and a SQL interface.

The offline storage tier

We currently maintain all the feature data on S3 in parquet format for the offline store tier. Data is stored at a feature group level and is partitioned on the feature, version, and date for easier querying. We also maintain a de-duped copy of feature values in delta format for quicker offline data processing in bulk/batch fashion.

To ensure consistent data in both the storage tiers, we use Kafka clusters as a front door. All data that needs to be written to the feature store can simply be published to Kafka. This setup gives us the flexibility to implement our consumers or use Kafka connect clusters to sink data to multiple data sources (online and offline stores). Thus, ensuring data availability in both stores and no inconsistencies in data across the online vs offline storage tiers.

How data is written to all storage tiers

Serving

The serving layer is available in two parts — a low latency scalable REST API service and an SDK written in Python and Java.

In tandem with our registry service, the REST API is aware of which feature is stored in which datastore. Based on the features requested, the API service pulls the feature values from the online store and serves the result to the user. The serving layer we built is currently a spring-boot application backed by HikariCP for connection pooling that helps us achieve good throughput in terms of requests per second.

Our SDK for Data Feast is another touchpoint to retrieve data from the feature storage. The SDK is written in Python and Java to cater to all codebases. These also come with support for spark Java/Scala and Pyspark). There are easy-to-use functions and methods to interact with Data Feast for data storage and retrieval.

How feature data is served for training and inferencing

Registry

A registry is the most critical component for any feature store. Keeping this in mind, we build a REST API service written using spring boot which is deployed in a high availability mode. An RDS instance backs the Service for metadata storage. We maintain all metadata related to features like name, description, owner, domain, team, storage tier enabled (offline/online or both), versions, data types, and other operations-related data.

The metadata in the registry sets the tone for Data Feast to function. We have automated jobs that use the registry to schedule and configure data ingestion and storage of features. The serving APIs utilise the registry to get information on which feature values are available, where to find them and how to access these features.

The registry service exposes API for FeatureGroup/Feature addition, edition, deletion, metadata retrieval and maintains a source of truth for all feature-related metadata. It helps us make the entire feature registration and metadata retrieval process as self-serve as possible. We also use the confluent schema registry to maintain the schema of data flow to and from Kafka. Additionally, the Data Feast registry service is responsible for other operational tasks such as registering feature schemas with the confluent schema registry, Kafka topic creation, and sink connector (Kafka connect) addition.

Monitoring and Alerting

There are two major categories of metrics that we need to capture to know the state and health of a feature store. First is the operational metrics about the production system as a whole, and the second is data-related metrics like quality, freshness, drift, etc.

As a first iteration, we focused on the operational metrics for the system’s core functionality.

We can track metrics related to Storage(availability, capacity, utilization, etc.), Feature serving (throughput, latency, error rates, etc.), and ingestion of feature data into the respective store (job success rate, throughput, processing lag, etc.) and more. We have thresholds set on these metrics that raise an incident for action to be taken when breached.

We are currently building a component to start capturing metrics related to data like data quality, tracking for drift, training-serving skew, and data freshness and profiling.

Transformation

During our initial phase of building Data Feast, there were a lot of existing data pipelines that created feature values already in production. Since the beginning, it was crucial for us to slowly start migrating the older pipelines to Data Feast and ensure all new pipelines get incorporated with Data Feast. To achieve seamless integration of feature engineering and computation, we decided to use Data Titan as the transformation layer for Data Feast.

Data Titan is a compute abstraction layer product that we built entirely in-house. It allows us to run data computation across various data stores with an option to run federated queries. We can decide on multiple destinations for the computed data, Data Feast being one of them. Data Titan is available in two forms at Dream11 — Batch and Real-time.

As the name suggests, Data Titan Batch allows computation on batch data from various sources like the data lake or different data warehouses. Data Titan is intelligent enough to understand where the data resides and directly run the computation on the data source. It also uses Apache Spark to run federated queries based on data sources and destinations.

Data Titan real-time lets us run a computation on real-time data sources like Kafka and store the data into Data Feast. Apache Flink backs this system for all immediate computational needs.

We will soon be releasing another blog to talk more about DataTitan and how it works, so stay tuned!

The high-level architecture diagram shown below brings the system’s current state together.

How all the pieces(storage, serving, registry, SDK’s) fit together to create Data Feast

The future of Data Feast

At Dream11, we have just begun our journey in the world of Feature Stores, and there is a long way to go. Here are some significant milestones that we have envisioned for the immediate future:

Integration of Delta or Apache Hudi format for time-travel capabilities on the feature data: We are currently evaluating both for feasibility and other advantages they provide.

Integration of Data Feast with Data Titan: The purpose is to provide an easy and seamless experience of generating features for the end-user.

Tackling issues proactively: We are working on building a feature data monitoring, profiling, and alerting component to tackle and catch any data related issues proactively

Further perfecting Data Feast: The feature store and data space are ever-evolving, and there is always scope for improvement at every step at any given moment. We continuously work to evolve our systems for scale and performance as Dream11 grows.

Are you interested in solving problems related to data at a multi petabytes scale, handling and processing billions of data points per day or working with passionate and innovative minds on the turf? We are currently hiring across all levels! Apply here to join us. More exciting stuff to come from the Data Team at Dream11! Stay tuned!

Reference Links:

https://hbase.apache.org/

https://phoenix.apache.org/

https://kafka.apache.org/

https://www.confluent.io/product/confluent-connectors/

https://www.rondb.com/

Data Feast — A Highly Scalable Feature Store at Dream11 was originally published in Dream11 Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Deployment At Scale: Story Behind Dream11’s In-House Blue-Green Deployment Platform ‘OneClick’.

Dream11 Engineering — Fri, 07 Jan 2022 10:00:05 GMT

~By Ritesh Sharma & Lijo Abraham.

Introduction

As a part of the agile-based development revolution, organizations have believed in rolling small and quick changes into the market. The theory of building fast-shipping fast plays a significant role in upgrading any product by rolling the features with very little time to market. However, this burdens the tech team’s efficiency in allocating dedicated human resources to ensure these changes seamlessly live onto the application on a frequent basis.

The answer to the referred problem was to automate the deployment process with the assurance of minimized failures and controlled rollbacks, let’s understand how we are achieving this using our in-house platform “OneClick”.

Seamless Blue-Green Deployment

Considering the amount of criticality due to the high scale on which Dream11 operates, we preferred to opt for the Seamless Blue-Green Deployment as our artifact deployment strategy to minimize the risk of downtime. This strategy of deployment is an extended version of default or instantaneous Blue-Green Deployment where a stack with new application changes faces the incoming traffic gradually unlike instantaneous Blue-Green Deployment.

Blue-Green Deployment — Approaches

Now let’s quickly understand our custom automated workflow for the Seamless Blue-Green Deployment.

Deployment Workflow

Briefly, this architecture mainly depicts the following components,

Jenkins: This acts as an Interaction layer for the deployment process and will obtain the information from users like targeted service name, how many shards are needed in this specific deployment, and a description of the change you are deploying for the service.
OneClick: This acts as a brain behind our deployments by taking the decisions of selecting the eligible infrastructure for deployments and performing smart traffic movement to achieve Seamless Blue-Green Deployment. (Stay tuned! We will talk more about this very soon in the upcoming sections below)
Concurrency Model: Our data science team at Dream11 has developed a model for predicting user concurrency using XGboost mode after trying multiple models with 100’s of features, to predict the half-hourly concurrency on the Dream11 platform (more details are here), So typically OneClick will use this information of upcoming traffic, and accordingly it will estimate the needed infra resources for deployment and respective Green stack creation calls can be made to Terraform.

Before we talk more about how OneClick Engine is achieving the Seamless Blue-Green Deployment, let’s have a glance at the typical microservice architecture in Dream11.

Dream11 -Typical Microservice Architecture

The above architecture depicts the incoming requests that will be interacting with cloud-managed DNS i.e. Route53 via Cloudfront. At Route53, load balancers are enlisted as weighted record entries. Here these weights determine the proportion of incoming traffic that should flow to the respective load balancers.

As Blue-Green traffic switch can be done at web-server or load balancer or DNS level. We preferred the DNS level switch over Loadbalancer’s due to the limitation of Targets per load balancer quota by AWS i.e. 1000 EC2 / LB, wherein peak intervals our most of the microservices exceeds these per load balancer limits and we need to take help of sharding in that case, to deliver the needed infrastructure resources by microservices (sharding is a process of distributing the traffic to multiple blue/active load balancers ).

Now let’s take a deep dive into “OneClick”. Below mentioned workflow represents the responsibilities and features of the “OneClick“ Engine which led us to achieve automated Seamless Blue-Green Deployment.

OneClick Workflow

🚀 Identify Eligible Shards

In this step, OneClick identifies the right set of needed load balancers out of the available load balancer’s inventory for microservice as per the requested number of shards by the users. Load balancers are categorized into Active (Green), Passive, and Eligible (Green) Loadbalancers sections along with their attached resources (Target Groups, ASG, EC2, etc ) and compiled into JSON objects. These Compiled JSONs will be processed for stack creation during deployments and traffic routing process and eventually shipped to AWS S3 for references during rollback events for Blue stack creation again.

🚀 Pre-Deployment Cleanup

In this step, the framework identifies if any stale resources (EC2, Launch templates, or AutoScalingGroups ) are attached to the eligible Green shards as a part of earlier deployments, and then clean up is performed for the same. Once the cleanup is done, eligible Green load balancers are blank entities with no other resources attached to them and ready to get utilized by new artifacts.

🚀 Prewarm Eligible (Green) Load Balancers

Since We perform the traffic switch (release) on the Green load balancers, during our peak traffic as well and hence we need to prewarm these load balancers on a prior basis to make them ready for serving the upcoming huge flash traffic to avoid any kind of load balancer capacity based anomalies. Prewarming of the load balancers done in the form of LCU (Load Balancer Capacity) units and its consumption is directly proportional to the results of function_max(new connections, active connections, rule evaluation, processed bytes).

As Dream11 traffic is highly volatile, the million-dollar question here is, how many LCUs are enough LCUs so, OneClick decides that using simple mathematics as mentioned below,

Eligible LCU Prewam Units = max(Minimum benchmarked LCU for micro-service, Current Provisioned LCUs On Blue LB, Consumed LCUs On Blue LB ).

Scale Up and Scale Down requirements for LCUs during Deployments get catered via infrastructure locks between our scaling platform i.e Scaler and OneClick.

🚀 ML-Based Resource Estimation

Thanks to our ML-based concurrency prediction model, which always tells us about the incoming concurrency on a prior basis for the Dream11 Platform. On the basis of predicted concurrency, our Scaler’s backend is able to generate the required number of EC2s that are needed by the deployment targetted micro-service so, OneClick simply just asks for needed EC2s from our Scaler backend for predicted concurrency and orchestrate the provisioning process.

We are working to tune our LCU estimation logic as similar to EC2 as mentioned above for more accurate estimations.

🚀 Green ( Passive ) Stack Creation

Once the estimation is done for needed EC2 (and other computing resources ) for deployment as per expected traffic, OneClick calls to our Terraform modules to procure the estimated capacity and attach it to Green eligible load balancers identified earlier by OneClick.

🚀 Pre-Route Operations

In this phase, we ensure the readiness of the Green Stack via,

Health Check Validations.
Ensuring, Green Stack capacities are not lesser than current Blue Stack in any case.
Acquiring, Inter-Automation Infrastructure Locks.

Let’s understand a little more about the latter one,

🚀 Inter-Automation Infrastructure Locks

As our infrastructure is growing so our automation entities are as well. These automation entities are responsible for doing some or other CRUD operations within the Dream11 infrastructure; With a growing number of invokers for the infrastructure operations, we needed a well-communicated approach within themselves and we are achieving this via a locking mechanism on Infra mutation activities. So while the deployment process OneClick acquires the lock-on microservice’s infrastructure and releases it post a successful traffic routing process and notifies other automation components for their infra-modification operations.

🚀 Seamless Blue-Green Traffic Switch

Considering all the pre-requisites and validations are done and there are no red flags, OneClick opens the traffic gates for Green load balancers. OneClick ensures this traffic shifts by moving some small percentage of total active traffic to the Green stack and observes a few verification factors to analyze any failures. These verification factors are nothing but the performance and functional metric (Load balancer Hardware Errors, Application Failures, Latency, etc ). Once captured metrics are within the defined threshold for microservice, OneClick sends more percentage of traffic to Green Stack and reduces the same portion of traffic from Blue stack, and reiterates the verification process. This iteration will keep continuing till Green Stack has 100% of incoming traffic and the previous Blue stack has 0% of the same.
During these seamless Blue-Green traffic switches if anything goes wrong at any iterations then traffic movement pauses and OneClick prompts the user for auto or manual rollback.

🚀 Post-Deployment Downscaling

Post successful traffic switch previous Blue stack is carrying 0% of active traffic and attached unused computing resources. These resources can be considered as wastage, hence OneClick downscales the capacity of attached AutoScalingGroups to Zero. By downscaling the Green stack completely and spawning quickly whenever it’s needed back, OneClick is helping us to fix one of the disadvantages of Blue-Green Deployment i.e bearing a high cost due to the approach of carrying the parallel stack in production.

Now the question is how we are managing deployment uncertainty of rollback decisions taken after a few hours or days from the deployment completion.

🚀 Rollback Pre-Validations

OneClick allows the users to take rollback decisions and moves traffic automatically back to the previous Blue (Active) Stack. In an earlier stage of deployments, OneClick has sent all the pre-deployment metadata of the Blue and Green stack to AWS S3, now OneClick fetches that metadata by unique deployment identifier and re-procures the infrastructure as per metadata. OneClick also validates the healthiness of the re-spawned stack and provides sign-off for infrastructure readiness before switching the traffic back to the previously Blue stack.

🚀 Rollback

After successful rollback pre-validations, OneClick seeks approval from the user, and on affirmative response, it goes ahead and quickly switches the traffic in a single iteration so, previous Blue (active) will be carrying the 100% incoming traffic again and the previous Green stack will be serving 0% of traffic.

The in-house OneClick engine has not only increased the confidence metric for our releases through inter-step verifications and quick rollback options but has also allowed us to leverage the Blue-Green approach without worrying about the cost of a parallel running stack. Similar to OneClick our Site Reliability Engineering team is continuously working to build more automation and reliability platforms that can cater to the need of Dream11’s high scale and operational needs.

That’s it from our end for now! Keep watching this space for more of our upcoming blogs, till then Happy Coding!

Deployment At Scale: Story Behind Dream11’s In-House Blue-Green Deployment Platform ‘OneClick’. was originally published in Dream11 Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Observability at Scale: How we built a cutting-edge Dream11 monitoring ecosystem ?

Dream11 Engineering — Tue, 30 Nov 2021 11:12:00 GMT

Observability at scale: How we built a cutting-edge Dream11 monitoring ecosystem ?

~By Umesh Gadage (SRE) , Mihir Shah (SRE)

As the world’s largest fantasy sports platform, Dream11 has over 120 million sports fans participating in exciting fantasy sports contests to showcase their skill and knowledge in various sports! With so many users using the app at a given time, our traffic pattern can get very spiky in nature. It can go from thousands of concurrent users to millions in just a few minutes! To support this dynamic pattern, we use prediction-based scaling and scale-out infra well before the match.

The Indian Premier League (IPL) is one of the biggest sports events in India and possibly the world that is viewed by over 400 million cricket fans. Every season, we scale to handle a peak of about 100 million requests per minute (RPM) at edge services. All the microservices generate many Application Performance Monitoring (APM) traces, system metrics, network metrics, DNS metrics, and more. These in turn, generate around 50+ TeraBytes of data and 65+ billion ingested spans per day.

Our Scale At A Glance -

Dream11 Scale

What were our challenges and how did we solve them?

Observability Lens from a single dashboard:

Monitoring a large infrastructure can be a difficult task. There are many moving components involved — from computing and networking, to application and business performance indicators. With a ‘top dashboard’ idea, only events that require attention and are actionable can be monitored during a fantasy sports contest. For example, the top API with response time of more than 200ms , a top API with 5xx, 4xx error of more than.01% , top RDS based on connection established or the CPU. This way, we created a ‘bird eye’ view of Dream11’s whole infrastructure within a single dashboard.

Using this bird’s-eye view of our footprint, we were able to quickly navigate through the issues and reduce the Mean Time To Detect (MTTD) and Mean Time To Resolution (MTTR). Our monitoring tool can build relationships between Cloudwatch metrics, APM metrics, network metrics as well as logs. We are easily able to navigate all of these, using a click of a button.

Bird’s-eye view of the critical metrics

CloudWatch metrics lag and API rate limit issue:

Initially, we were sending metrics to our monitoring tool using an Amazon Web Services (AWS) integration. Our monitoring tool scraps CloudWatch data and stores it as an integration metric. During big events, we generate over 700 million Cloudwatch metrics per day. The AWS integration with the monitoring tool scraps all the metrics from Cloudwatch and sends it to the platform on a configurable regular interval of two to five minutes. Due to the huge volume of metrics, we end up with an API rate limit issue at the AWS end and also an approximate delay of 20 to 30 minutes in metrics. This leads to a lot of false and late alerts. AWS has come up with a CloudWatch Metric Streams solution to handle such scenarios. Using these streams, we can send metrics to any major monitoring solution provider in almost real-time with an open telemetry collector. After implementing this solution, the delay was reduced to almost one to two seconds which is negligible, and the improvement is significant with cost benefits. AWS Cloudwatch MetricStreamUsage is almost four times cheaper than AWS Cloudwatch GWD-Metrics ( Scrap ).

Scrap GWD Metrics

Cloudwath Metrics Stream

Network Monitoring:

At Dream11 scale, network monitoring plays one of the most critical roles in the troubleshooting of our systems since our network is complex and distributed across microservices. Our monitoring tools help us in keeping track of our network Ingress/Egress architecture alongside the application, infrastructure, and DNS performance for faster troubleshooting.

Sometimes, we face intermittent issues with the AWS Network in one particular availability zone. Using our network monitoring, we quickly get to know if there is any abnormal activity in our network. A clear signal to validate, is to check the TCP retransmits by AZ. We have multiple slice-and-dice options available to see our network performance. This has capabilities to filter traffic across the availability zones, services, ENVs, domains, hosts, IP, VPCs, ports, region, IP type, etc. (public/private/local).

Network Monitoring Metrics At Dream11

Upon observing these issues, we quickly removed the faulty AZ from our microservice Auto-Scaling Groups using custom in-house automation scripts. Since all our microservices have their own dedicated autoscaling group, removing one AZ can actually spin up new instances in another AZ using AZ Rebalance policy. Similarly, we removed all the faulty subnets from ALB/ELBs for that AZ.

Automatic Service Map:

Our APM tool provides end-to-end distributed tracing from frontend devices to databases — with no sampling. By seamlessly correlating distributed traces with frontend and backend data, our monitoring APM tool helps us to automatically monitor service dependencies, latency, and eliminate errors so that our users get the best possible experience. Distributed tracing is a technique that helps us address the problem of bringing visibility into the lifetime of a request across several systems which is very useful while debugging and figuring out where the application spends most of its time. We have a Service Map, where we can inspect all the services to get their RED metrics along with their dependencies and filter capabilities by Application Service, Databases, Caches, Lambda Functions and Custom Scripts. Also, this service map reflects almost real-time, and the monitoring agent sends data to our tool at an interval of 10 seconds. The service map shows all services in green when there is no issue, but turns red once anything goes wrong. This comes from the monitor ,which is configured for each service.

Automatic Service Map

Scale Verification:

We also conduct predictive scaling based on our data science model. For more details on how we scale, you can read our blog post here. The data science model at Dream11 predict the concurrency expected for each match. Based on this prediction and our load test, we benchmark our numbers to scale over 100 microservices. Previously, we used ASG scales manually and validated them on a regular basis, which was a very complex and time-consuming task. But now, this process has been automated and our monitoring tool helps us to verify much faster than before. We have set up a few alerts which help us to identify where we need to look to resolve issues.

We send expected capacity data from our benchmark for an hour and from a cloudwatch to our monitoring system. The comparison between the two helps us to detect provisioning issues faster.

Scaler Verification Dashboard

This is how we solved observability at our scale, to provide the best possible user experience to over 120 million sports fans! Keen to know more about how we solved complex problems and build our own solutions? Stay tuned for our upcoming blog posts on:

Cost Wastage Portal
Automated Scaling From data Science Prediction
Adoption of Graviton & Granulate
Migration from Gp2 → Gp3

We are hiring! Come, be a part of our dream team by applying here!

Turf- Designing the rules of play!

Dream11 Engineering — Tue, 23 Nov 2021 09:20:38 GMT

Introduction

One of the most common traits of a successful application is the consistency in the use of colours, shapes, and patterns that leads to brand identification. At Dream11, we aim to standardise our entire design practices and design components. Turf, our design system, ensures that all our 120+ million users experience the same brand value, whichever platform they may be on. For example, the color combination used repetitively is intended to evoke a brand identification subconsciously regardless of where they see it. Using the exact shades and combination is of paramount importance, which we achieve through our design system. Such elements distinguish a great product from a good one.

Why should a design system be implemented?

A design system aspires to create a shared vocabulary between designers and developers. It allows for a centralized design source where all design-related attributes and components can be stored. This helps maintain consistency in designs and ensures efficient design changes across platforms. It not only helps in minimizing developer interaction time for such changes but also reduces the possibility of human errors.

How did Dream11 work prior to implementing a design system?

Prior to working with a design system, we did not have a single source of truth. Each platform maintained their own copy of design values and components. There was no standardisation among user interface (UI) components. This led our teams working on different features to create the same components with different names, which caused redundancy. Also, since each platform maintained its own design source of truth, any changes in the design required the active involvement of developers.

Traditional flow of design values

What are the objectives of using a design system?

Forming platform-agnostic design values

The first step towards consistency will be creating a central and designated place for design elements that can be referred to by all teams. Developers and designers can collaborate to define the common vocabulary which will be the basis for all components and tokens used across designs. The design elements and components defined, should remain platform agnostic and act as a database of raw values. These raw values would then have to be processed in multiple steps for consumption by different platforms.

Adopted design values flow

Giving more control of the design changes to designers

Once the design system comes into the picture, all UI elements including colour, dimensions, fonts, etc., will be parameterized. This ensures that all parts of the platform that need a particular design element, use the same token. The value of the token should remain hidden from the developer. This will also ensure any changes made to the token value percolate into all parts of the platform without any extra effort. Moreover, since the source of truth is outside all platforms, it will enable designers to make design changes without any developer intervention.

Standardising design components

Once the design system is in place, it is expected to serve as a database for components. All components would be documented with code and property descriptions. This will help developers to know about existing components and thereby, minimise the possibility of duplication. Also, if any new components are required, they can make a component request which will then be created and documented for use by everyone. Moreover, these will be generalized enough to serve multiple use cases. The design elements within these components will be tokenised which will ensure all design changes to these components and other composite ones derived from these, will take place in a cascading manner.

The Turf

To meet our expectations, we architectured a multi-level system with well-defined interaction zones for designers and developers. This system allows for a single source of design values to be referred to by designer and developer software, with clear guidelines and toolings around it, so logging and communicating changes to design values can be done.

The philosophy of tokens

In its essence, we have adopted Brad Frost’s Atomic Design Pattern for our design elements. The smallest units in a design are attribute values like color, dimension, font, etc. These are treated as atoms of the design system, which then combine to form molecules. These, in turn, combine to form organisms and so on. These attribute values or tokens are theoretically design decisions translated into code. To increase flexibility and allowance for easy and safe usage, we have defined three levels of tokens — global, alias and component. Each level of abstraction helps in various levels of mapping design values.

Example of an atomic design pattern, Source : https://atomicdesign.bradfrost.com

Global Tokens

These are the primitive values in our design language, represented by context-agnostic names. They map directly to the absolute value of an attribute. The global tokens define the superset of values (colors, dimensions, fonts) that are permissible within the app. Global tokens follow one-on-one mapping with the attribute values, meaning, no two global tokens will point to the same attribute value.

For example, in the below diagram, a global token could be named color_global_group1__10 . This is context agnostic. Here, the palette is defined with a color holder, whose value is defined by the designer, which can change.

Example of global tokens

Alias Tokens

These are context-specific tokens that derive their values from global tokens. Unlike global, alias tokens have a many-to-one mapping, meaning multiple alias tokens might map to the same global token. These are generally consumed by different parts of the app.

For example: Alias tokens refer to a global token without knowing what value it holds. color_alias__group1Light is pointing to color_global_group1__10. Even if the value of the global token changes, there won’t be any changes needed in the alias tokens.

Example of alias tokens

Component Tokens

This is a special class of alias tokens that are specific to the components. Component tokens allow us to maintain a cleaner mapping for base components and higher-level composite components.

Tooling and processes

The main layers

Token Management

The token management layer serves the purpose of storing all design values and allows designers to implement their design decisions. The attribute values, theme details, component code, and descriptions are maintained in this layer and it serves as the source of truth from which all platforms and design software derive their design values. One thing to note is that all design values which are consumed by developers are stored in a platform-agnostic manner.

Token Management Layer

Transformer

This layer is responsible for consuming the platform-agnostic, raw design values from the token management layer, processing it, and generating platform-specific code. Special care is taken to ensure versioning of design values is done without any reprocessing of the raw values. The transformer layer is written to serve all platforms currently in use — Android, iOS, React Native, and Web.

Transformer Layer

The Processes

Processing tokens

Since raw design values are too unstructured for processing, we run a nightly process that converts these raw values into schema-based jsons. Separate jsons are generated on the basis of the category of attributes. This process happens inside the token management layer. The versioned jsons are stored separately for consumption by the transformer layer.

Real-time updates

To ensure that changes made to designs are communicated instantly, we have set up a real-time update system that can alert the concerned developers and designers. The alert can be sent via multiple channels like slack, mail, etc., and carry information pertaining to the type of modification, user involved, time and date, etc.

Design adoption

After the platform-specific code is available to developers, there is no guarantee that the developers will start using them. This might be due to multiple reasons, such as lack of proper understanding of the design system, not being aware of tokens and components that are available, and so on. As such, it becomes necessary that we provide a nudge to the developers to adopt the design system. Each platform has its own way of handling this.

In Android, we piggyback on the existing linting framework. A major reason to adopt Android Lint is that it can run static analysis on the code. This means we can check our code for design violations as we code without having to wait for compilation. In addition to the existing standard rules provided by Lint, we have created our own custom Lint module with rules to enforce the usage of our design system elements and components while flagging the default Android components.

Apart from the static analysis, we run all the Lint rules on our project every time a pull request is raised. The benefit of this is two-fold:

The development is not hindered by throwing lint errors during local compilations
Any code being merged to master is fully compliant with the design system.

In iOS, we follow the same principle to avoid hindering the development by not running any compile-time checks. Instead, a script runs during every pull request which flags any violations to our design principles.

What’s next?

We hope this blog post offers you a detailed overview of the philosophy, toolings, and processes in place for Dream11’s design system. We have more to come in this series on how we built up the design system from scratch, detailing the technical challenges we faced and how we resolved them. So, stay tuned!

Keen to work with the Dream11 design team? Apply here to join us!

Enhancing security and trust with AWS WAFv2

Dream11 Engineering — Fri, 24 Sep 2021 07:43:13 GMT

— By Parag Suchak, Ritesh Kumar

Web Application Firewall (WAF) is considered a critical part of an effective security solution that offers website security and strengthens the overall security posture. The end goal for any web application Firewall is to protect the application against:

Known vulnerabilities and threats
Identification of and protection against bad bots
Protection against Distributed Denial of Service (DDoS) attacks
Real-time intelligence to block blacklisted source
Geolocation Enforcement

As the world’s largest fantasy sports platform with 110 million+ users, we at Dream11 run multiple contests simultaneously while maintaining optimum user experience and efficiently process millions of user requests per minute. Our user-first and data-driven teams have made it their priority to ensure that the Dream11 app remains protected against common as well as advanced threats and vulnerabilities. In this blogpost, we will explain how AWS WAF has helped us to do so and why we switched from AWS WAF classic to AWS WAFv2 for better manageability and more visibility.

How does WAF work?

Figure: Working of a Web Application Firewall

Web Application Firewall is configured between client and web server. For every request from the client, WAF inspects and checks for any WAF rules that match to the requests. If a particular request matches the WAF rule. For example, in the Cross-Site-Scripting payload as shown in the diagram above, the request will be blocked by WAF and it will not reach the Application Server. WAF also supports putting specific rules into count mode/monitor mode where requests will not be blocked but they will be tagged differently to understand how the rules works.

AWS WAF is a prebuilt service provided for protecting applications that are hosted on the AWS cloud. It can be deployed on the following sources:

Amazon CloudFront distribution
Application Load Balancer
Amazon Application programming interface (API) Gateway REST API
AWS AppSync GraphQL API

AWS WAF comes with two different versions:

AWS WAF classic
AWS WAFv2 (Introduced in Nov 2019).

Why AWS WAFv2 ?

AWS WAF classic is powerful and can protect against common attacks. However, there are many improvements in AWS WAFv2 over the AWS WAF classic which add more powers to handle malicious attacks more efficiently.

Table: Resolving various challenges with WAF Classic and WAFv2

After considering all new enhancements offered in WAFv2, we decided to migrate our WAF classic to the new AWS WAFv2. There are two ways to do so.

Option 1: Using Migration Wizard

Option 2: Manually create new WAF rules & Conditions

The Challenges:

The migration wizard can migrate only unique rules across all WebACls. Also, it cannot migrate WebAcls across AWS Accounts
Manual migration requires setting up everything (including rules and conditions) from scratch, which was too complicated and time consuming for us.

Our Solution:

We took a hybrid approach which includes the use of the migration wizard and a little manual work to solve this problem.

First, we migrated all unique rules with the wizard across all the WebAcls. Then, we got the JSON code for common rules and imported JSON across all the WebAcls as per requirements. WAFv2 supports visual consoles as well as JSON editors for creating new WAF rules and conditions. We just required importing the JSON of the common rules and created new customized JSON rules across the AWS accounts.

Things to consider while migrating using Migration Wizard:

Only web ACL can be migrated within the same AWS account
If multiple web ACLs share the same resources like IP sets and regex, they will be duplicated under the new WAF
Rate-based rules can not be migrated
Managed rules from the AWS marketplace can not be migrated
Logging for web ACLs will be disabled by default
Configured cloudwatch alarms in WAF classic require the metric name modification
AWS WAF Security Automations lambda and functions need to be updated manually.

If you’d like to know more about AWS WAF, click here.

To join the Dream Team and work on exciting projects with us, click here.

Enhancing security and trust with AWS WAFv2 was originally published in Dream11 Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Lessons learned from running GraphQL at scale

Dream11 Engineering — Fri, 10 Sep 2021 11:09:31 GMT

— By Shrey Mehta, Kaushik Barodiya

At Dream11, we have experienced tremendous growth from just 300,000 users in 2015 to over 110 million users at present. To grow at this blazingly fast pace, we moved to a microservice architecture for developing backend systems.

As the number of microservices grew, it became increasingly difficult for frontend developers to fetch data from multiple services and present it on the UI (user interface). So, we decided to come up with a presentation layer that could provide us with:

Network aggregation
Data transformation
Type safety

After due deliberation, we decided to use GraphQL

GraphQL at Dream11

Brief History of GraphQL

Back in 2017 when GraphQL was just a buzzword and not many companies were using it at scale, at Dream11, we started using it in production. It was a simple expressJS based application that we deployed in front of our microservices based stack. We received positive feedback early on from our web teams and slowly moved our remaining clients — Android and iOS to GraphQL as well. At the same time, our active users were steeply rising. Our GraphQL usage doubled year on year.

Max. traffic on GraphQL year on year

Traffic Pattern

Typical traffic pattern

As seen in the graph above, there is a sudden burst of traffic during crucial events of a real match. During this time, traffic increases by almost 100% (e.g. 14 million to 28 million in 2 minutes at 20:00) in a matter of minutes. We use Amazon Web Services (AWS) at Dream11 and its components cannot auto-scale at this pace. So, we provision the load balancers and servers based on maximum predicted traffic. If you want to know more about how we do this, check this interesting blog that our SRE Dreamsters wrote on it.

Current Architecture

High-level architecture of GraphQL service at Dream11

GraphQL servers are running on compute-optimised eight-core machines (c5.2xlarge). Each Elastic Compute Cloud (EC2) is managed by Auto Scaling Group (ASG). We provision approximately 7,500 instances for 1 million requests per second.

To route the traffic to these instances, we are using multiple load balancers. Keeping multiple load balancers solves two problems for us:

A single load balancer can route traffic to only a limited number of instances.
A single load balancer can become a single point of failure.

Owing to the spiky traffic pattern, we have to pre-warm the load balancers as per the predicted peak traffic.

Optimisations Done Over The Years

To accommodate this kind of hyper-growth and impulsive traffic patterns, the GraphQL service went through multiple optimisations over the years. We used multiple strategies to optimise resource utilisation. Few things that really stood out were:

1. Batching

In GraphQL, each resolver function only knows about its own parent object. If a case arises where two resolvers are making requests for the same data, in order to avoid it, we use Dataloader, which is a library to batch requests.

Example: Duplicate network calls (given: matches and sites calls the same data source)

2. Caching

In order to reduce the load on graphQL servers, caching is done at two levels- one, on the graphQL servers, and two, on the clients.

To enable caching, each backend Application Programming Interface (API) sends X-TTL headers with the response. GraphQL caches globally cacheable resources for a certain amount of time (TTL).
During query evaluation, graphQL calculates the TTL for query response and sends it to clients. This enables clients to cache graphQL responses.

Impact On Downstream Services

GraphQL performs a lot of common operations for all microservices e.g. authentication, data manipulation for the presentation layer, data caching, etc. This allows microservices to be simple and decoupled from each other. Caching has reduced load on microservices and hence, we could aggressively downscale them.

Costly Abstraction

The Indian Premier League (IPL) is a celebrated sporting festival in the country and we see an enormous amount of activity on the Dream11 app. The League’s 2020 edition was special for us since Dream11 was the Title Sponsor. During the League, we received peak traffic of around 40 million requests per minute with a peak concurrency of over 5.5 million. In spite of performance optimisations being done year after year, we heavily relied on the increased capacity to provide 100% uptime, which resulted in very costly infrastructure.

The graph shows the cost of GraphQL servers during the IPL 2021 with respect to other microservices

service wise aggregated infrastructure costs for IPL 2020

For IPL 2021 we predicted almost double the traffic that we received in the previous edition(IPL 2020).

What were the main challenges?

While running GraphQL on production we faced two major challenges:

1. Unpredictable Performance

Previously there were random spikes in CPU (Central Processing Unit) and memory utilisation, owing to a number of reasons :

Cache TTL expired on the clients
Garbage collection
Resource heavy query.

2. Zero Tolerance For Failure

GraphQL is at the edge of all the business-critical microservices, so it has to be up and running 100% of the time in order for Dream11 to work. No risk can be taken on this layer so it has to be provisioned for peak traffic for any event. This led to high computing costs for the system.

As traffic increased year on year and we kept adding more features on GraphQL, infrastructure requirements also grew.

In the IPL 2020 final, we had a footprint of 13k instances deployed behind 13 load balancers.

With such a large fleet of EC2 instances, the deployment becomes very complex and time-consuming, and experimentation becomes nearly impossible. This made us relook at the whole architecture again and further optimise the GraphQL layer.

How to measure performance?

“ We can only optimise what we can measure ”

Defining the right metrics to measure performance and identifying correct SLAs on those metrics is one of the critical parts of the optimisation process. Typically performance is measured in terms of latency, throughput, and resource (CPU, memory, and network) utilisation. Having the right tools to measure these metrics quickly is a prerequisite to optimising performance.

Synthetic benchmarking

Doing a load test on a production-grade GraphQL server is very expensive in terms of cost and time. So, we created a synthetic environment that mocked the backend services and generated a dummy load on the GraphQL server.

Tooling

1. Mock API server

We created a mock server to mock API responses from microservices with simulated latency.

2. Custom HTTP client

We have created a custom HTTP load client to simulate the traffic pattern that we get on our production servers.

3. AWS Cloud Watch and Datadog

We monitored the CPU, memory, and latency of the servers using these tools.

Defining Metric For Performance

Defining the correct metric to improve is a very critical part of the optimisation process. For us the goal was to improve the max-throughput ie. the maximum number of requests the system can handle in a second with p50 latency under 200 ms

Upon benchmarking the most commonly invoked queries on a single graphQL server in a load environment, we found out that throughput varied a lot from one query to another, for some of the queries throughput was as low as 50 RPS and for others, it was as high as 500 RPS.

This gave us baseline performance of each of the queries, we could then quantitatively measure any performance optimisations that we would do.

Production Benchmarking

Synthetic benchmarks are good to quickly measure performance relatively across different branches. But the production environment system is much more complex, so we could only extrapolate synthetic benchmarks on production to only a certain extent. To measure performance gain on production we used the canary deployment technique to deploy any performance optimisations.

Production Performance Metrics

We wanted to define a performance metric that we could track on production. But on a distributed system where a number of instances keep on changing and request rates also vary a lot over time, to compute performance improvement we defined a custom metric as

Definition of Work Done

Definition of Performance

Finding Out What’s Going Wrong Under The Hood? 🧐

In order to find out what’s causing the performance issue in GraphQL, we used the following approaches.

Profiling

We started by profiling one of the production servers during peak traffic with the help of the pm2 enterprise version profiling tool. It allows you to download CPU and memory profiles for a particular process which further can be analyzed using various tools like chrome performance tool, vs code, etc.

CPU Profile

On analysis of CPU profile, we found out that

On average 20% of total CPU time was being spent by garbage collectors.
GraphQL validate operation was taking a 3% of CPU time
Some of the resolver code was long-running

CPU profile of GraphQL server during load test

GC profile

In CPU profile analysis we found out that GC was contributing majorly to CPU time, so we wanted to analyze GC further. To analyze GC we used these tools and figured that scavenger GC was happening too frequently.

Inline Caching Heuristic

V8 engine performs a lot of run time optimisation based on certain heuristics. Inline caching is the crucial element to making JavaScript run fast.

We used Deoptigate to identify the hot code which was not being optimised by the v8 engine. We found out a few of the utils functions were megamorphic and were not being optimised by v8.

Analyse APM (Datadog) data

dd-trace (Datadog library for nodeJS) automatically instruments graphQL module. It provides APM data based on queries. Based on this we were able to spot frequently invoked queries.

In order to gauge the complexity of a query, we track the time taken by graphql’s execute operation and the average number of HTTP calls made inside a query.

Datadog UI Query wise APM data

Internal Metric of a Query

Micro benchmarking

We benchmarked each widely used library function or pattern against their alternatives using Benchmark.js. Some of the comparisons are described in the next section.

Learnings

With dataloader and multi-level caching in place, the network was already very optimised.
Based on our load test results we could see that CPU was the bottleneck for us.
There were no low-hanging fruits.

Approaches for optimising performance

With all the profiling and benchmarking tools handy we started making small changes and measuring them for performance gain.

1. GC Tuning

In our GC profile analysis, we found that for 3 minutes of load testing,

the server was spending 30.6 seconds in GC.
On further analysis, we found that during this period 556 scavenger collections were performed.

Based on this we could see that the problem of the GC is concentrated in the Scavenge collection stage, that is, the memory collection of the Young Generation. The GraphQL server was generating a large number of small objects in Young Generation Space. And hence, triggering scavenge collection. In this way, the problem boiled down to optimising the Young generation to improve application performance.

In nodeJS, the size of young generation space is determined by the flag `— max-semi-space-size` which defaults to 16MB. We tried increasing the max-semi-space-size value with values as 128MB, 256MB, and 512MB. We did a load test with all these values and noticed that at 256MB system was at peak performance. After deploying this optimisation on production CPU utilisation went down by 12%.

Load test results at different max-semi-space-size

2. Fastify Web Server

Fastify is a fast and low overhead web framework designed for NodeJS. On public benchmark, it promises to be 5 times faster than the Express framework. Since all the plugins/middleware used were portable we experimented with Fastify. On production, we got 10% reduction in CPU utilisation.

3. R.curry vs Uncurried

To improve the compositionality of our code we had defined a ton of curried functions using Ramda. But every abstraction has a cost associated with it. On profiling, we found out that the `curry` function was taking up a lot of CPU.

Two versions of the add function

Benchmark result add vs addCurried

Benchmarks show that removing curry from the code makes it up to 100 times faster. Curry function is at the core of Ramda, almost every function is curried. So from this, we came to the conclusion that Ramda is becoming a performance hog for us.

4. Immutability vs Mutability

Two ways of adding a key to an object

Benchmark mutable vs immutable

In our code, we had written immutable code everywhere. Looking at this benchmark we decided to remove immutability from the hot code.

5. Monomorphic vs Polymorphic vs Megamorphic Functions

V8 engine optimises monomorphic and polymorphic functions at the run time making them way faster than megamorphic functions. We converted a few frequently invoked megamorphic functions to multiple monomorphic functions and observed performance gain.

Megamorphic to monomorphic conversion

6. Lazy Evaluation

In GraphQL a resolver can either be an expression evaluating a value or a function that returns a value (or a promise of value). Resolver expressions are eagerly evaluated irrespective of fields invoked in any query.

GraphQL schema

Eager resolver

In the above example if the client queries only id and title then also groupPlayerByType will be executed. In order to prevent such unnecessary invocations, we can wrap these operations inside a function.

Lazy Resolver

This will ensure that groupPlayerByType will be called only when groupedPlayers is queried.

7. Caching query parsing and validations

Graphql performs 3 steps to evaluate the result of each query.

Parse (Creates AST from the query)
Validate (validate AST against the schema)
Execute (Recursively execute the resolvers)

From the CPU profile, we found out that 26% of the CPU time was spent in the validation phase. The server was doing query parsing and validation for every request

CPU profile of GraphQL server during the load test

But on production, we get requests for a limited set of queries so we could actually parse the query and validate it once for each query type and cache them. In this way, we were able to skip the redundant parsing and validation steps for subsequent queries.

8. Infrastructure Tuning

All requests coming to `https://www.dream11.com/graphql` get routed to multiple load balancers using weighted DNS. Weighted DNS doesn’t guarantee the exact distribution of requests because of DNS caching at the client-side i.e requests coming from a client will go to the same load balancer for a particular period of time (DNS TTL). So even if we assign equal weights to all load balancers there would be some load balancers that would get extra requests, which puts extra load on instances behind the load balancers.

This variance in distribution is directly proportional to the number of load balancers and DNS TTL.

After all the optimisation we were able to reduce the number of servers that allowed us to reduce the number of load balancers.
We tuned the TTL value to reduce the variance in traffic distribution.

Results

We started the optimisation project in November 2020 and had only 5 months to figure out performance hogs, optimise, test, deploy and scale down before IPL 2021. We did massive refactors throughout the project and did multiple deployments in this period. Following are some of the results:

Latency

In IPL 2021, overall p95 latency was reduced by 50% in comparison to IPL 2020 which resulted in a better user experience and allowed us to reduce infrastructure footprint further.

P95 latency graph of GraphQL

GQL time

Average GraphQL execution time is reduced by 70%

GraphQL Execute time

Average Performance

We tracked average performance week on week and after all the optimisations and infrastructure tuning it was improved by more than 50%.

A week on week MIS data plot

Relative Cost

From being the most expensive service by a very high margin, GraphQL is now comparable to other services in terms of cost.

AWS Infrastructure cost of services at Dream11

Instance Count

GraphQL production instance count for serving 5M concurrent users

After doing lots of micro-optimisation and some infrastructure tuning. We were able to serve 5M concurrency with 80% fewer instances in the first half of IPL 2021 as compared to IPL 2020. Our daily average cost is reduced by 60% and projecting a similar trend will help us to save more than 1 million dollars during IPL 2021.

Key Takeaways

“ Aggregation of Marginal Gains ”

When we started we couldn’t find any low-hanging fruits. We approached our target of reducing infrastructure from the first principle. We questioned every decision which was taken right from choosing NodeJS as the stack to AWS instance type. We did load tests for the slightest improvement we could think of. Some of the key takeaways from this project would be:

Instrumenting efficient ways to measure performance is key for optimising performance.
Multiple optimisations of small improvements can give you larger aggregated performance improvements.
At the edge layer, SLA for latency can be relaxed.

If you are interested in solving complex engineering problems at scale, join the Dream11 team by applying here.

Lessons learned from running GraphQL at scale was originally published in Dream11 Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Don’t let that consistent launch time slip away!

Dream11 Engineering — Thu, 26 Aug 2021 16:28:38 GMT

— By Hardik Parmar, Kunal Chavan, Rupesh Urmaliya

Introduction

Every day on the Dream11 app, we host thousands of contests so our 110+ million fantasy sports fans can engage with their favorite sports in the palm of their hands. Since the team lineups are usually announced around 30 minutes before the real-life match begins, many of our users choose to join contests moments before the match start time. This is where the need for an optimum user experience becomes crucial for our success. For any app, every aspect of its performance is a significant factor that could boost user experience and it’s the same for the Dream11 app too! And, one important factor affecting the app’s performance and ultimately the user experience is the launch time.

Why is launch time crucial?

The first impression of an app can be a deciding factor for the user. And the very first impression that the user forms about an app, is the time it takes to launch. A high launch time exponentially decreases the user’s happiness quotient. Besides, the system may kill the app for being unresponsive for too long. Hence with an increase in the launch time, we may lose the valued user in the process and that can affect the overall business metrics.

Why is it critical for the Dream11 app?

For us at Dream11, it is no surprise that the fill rate for most of the contests on the app increases during the last 30 minutes prior to the deadline. At such a crucial time, the delayed launch of the app would result in failure to meet this deadline. If the contests are left with unfilled spots, it could lead to not only a poor experience for users but also a loss in revenue. Hence, an optimized launch time is very critical to avoid this.

Consistent Launch Time

Maintaining consistent launch time

While developing a feature, we might think that a small increase in the launch time would not affect the overall user experience. When we actually compile a release with multiple features, the collective increase in the launch time can be significantly higher. Users have a varied number of devices with different operating systems and with different network bandwidths. A small increase in launch time for one device with a given configuration might be significantly higher in a lower-end device with a lower configuration. Also, we have continuous weekly releases with new features, some of which are even developed by new developers. All of this could make maintaining a consistent launch time of the app difficult.

The Challenges

Earlier, we saw how a cumulative increase in launch time becomes a barrier to the user from joining the contests, especially near to the deadline, resulting in unfilled contests. This situation could become even more severe when the launch time is not maintained and a sudden spike occurs in a crucial release just before a big tournament (God forbid, during the IPL🤞)! Hence, we knew we had to keep an eye on the launch time and maintain its consistency.

Our trials and errors

To maintain the consistent launch time, we initially used Firebase performance monitoring (Android) and MetricKit (iOS) to capture the app’s launch time post the app release. The metrics could be accessed using the Firebase dashboard and MetricsKit in XCode.

Why did we need an upgrade over these tools?

Firebase and MetricKit took hours to reflect the latest data
The data was only available after the app release.

As our team size continued to grow, sometimes, a typical app release would have a bunch of new features going live. This impacted the launch time drastically, which could or could not be optimized before going live or until the next app release. Hence, there was a need for a tool that could provide launch time insights before the app was released.

Our Solution

While we tried to figure out how to tackle this, we thought about creating an automated process that could help us keep a check on the app launch time on a regular basis.

The basic workflow we came up with goes like this:

Build the app and launch it multiple times on a simulator/emulator
Log the launch time and derive a conclusion based on the cumulative launch time data

The Context

Measuring The App Launch Time

Let’s understand how we can measure launch time on both Android and iOS platforms.

Android launch time

When a user clicks on an app icon launcher, the app process makes an inter-process communication (IPC) call with the ActivityTaskManagerService.startActivity(). Then, the system server process sends a request to the Zygote process, to fork a new process for the app. Before creating a new process for the app, the system server process creates a window that will be visible to the user until the app draws its first frame. This duration is called the ‘app launch time’.

We can learn more about the Android launch time from the official docs here.

iOS launch time

There are two phases of launch time in iOS — the pre-main and the post-main.

The pre-main

The dynamic linker (DYLD) statistics can really help in capturing the pre-main time. This is a time-measurement that starts when the app’s main method is called, till it gets control from the operating system (OS).

We can get the pre-main time from Xcode. The DYLD has a built-in environment variable to get this measurement.

Once the variable is set, we get the pre-main time after running the app in Xcode’s console. The log looks something like the below image:

The post-main

Post-main is the time taken by an app from UIApplicationMain to the first screen presented to the user. In short, it is the time taken from AppDelegate/Main to the first frame that the app renders.

The actual launch time is the sum of the pre-main and the post-main.

The Implementation

Now that we know how we could measure the app launch time, let’s talk about the solution we built to create the above described automated process of measuring the launch time.

Tooling & Instrumentation

Let us take a look at each step of the automated process and understand the usage of various tools we used to achieve this:

Automation

We used fastlane and Jenkins to automate the entire process:

We created a job on Jenkins that includes various configurations i.e. app launch count, git branch, device configurations, etc.
We configured the job to trigger periodically at regular intervals
We used fastlane to run our custom scripts for the workflow by creating separate lanes.

This automated trigger would kickstart the process of building the app once and launching it multiple times on configured devices to collect statistically significant data.

Collecting Data

Once the job was triggered, Fastlane would run our scripts that are specific to both platforms. Let’s talk about these separately.

Android

We integrated Firebase performance monitoring tools in our Android app which gave us the data we needed to track the launch time.

Let us go step-by-step on how we retrieved this data:

We used a Kotlin script that installed the app on the emulator, launched it multiple times, and captured the logs exposed by Firebase SDK

We stored these logs in a given log path directory.

iOS

We wrote a similar script for launching the simulator, installing and running the app on it. Once the app was launched, we wrote the time taken until our first view loads into a temporary file. Again, we did this multiple times until we collected a statistically significant amount of data.

Let us take a close look at the process, step-by-step:

Launched the iOS simulator from the Xcode directory
Built the app using fastlane’s xcodebuild command
Installed the app in the simulator using xcrun

The following code would calculate the time the CPU took to launch the app and present the first screen to the user

To get the best results, we ran this code on the first call of `applicationDidBecomeActive(_: UIApplication)`, and stored this time in a separate file.

Data Analysis

To collect significant data we used these scripts to launch the app multiple times and gather statistical data. This raw data needed to be processed further to help us derive a conclusion.

To do so, we derived the following data points from the collective data:

Minimum
Maximum
Average
Median

We analyzed these data points for each job result and made sure the launch time was maintained properly.

Maintaining Historical Metrics

To make sure that the launch time was in the acceptable range, we stored all the launch time data on a remote database and queried it via Kibana.

Additionally, we posted all these data points to a Slack channel, to get instant alerts when the job was finished.

Process changes

We changed our release process to only merge the features that passed the acceptable threshold of the launch time.

Since the releases were very frequent, it was important to reveal the launch time effects of a feature we built well before the release. Hence, any improvements if necessary could be made without affecting the timelines.

The Impact

The major impact of our solution was that the launch time was consistent and stayed within the acceptable range which was decided prior to the release.

This was only possible because the launch time data was available to us even before the release. This also helped us in identifying any anomalies and rectify quickly while keeping both user experience and timelines intact.

For a data-obsessed team like us, data is everything. Having the launch time data available prior to release increased our release confidence and decreased the overall turnaround time of measuring the launch time for a particular release.

Our Next Steps

We have definitely solved our initial problem of maintaining launch time before the release. However, there are some areas which could be further improved.

Current challenges

The major problem with our current system is that we manually look at the available data and decide if the launch time falls under the acceptable range. This introduces a chance of human error. We want this to be an automated process and remove the human component as much as possible.

Proposed solutions

Since we have a dedicated Pull Request for each feature, it would make sense to create an alert system for each PR which would get triggered on each commit and run the job. If the launch time for the PR is out of the acceptable range, it should give us an alert, preferably on Slack.

These improvements should reduce the turnaround time significantly and help us automate most of the tasks.

Conclusion

Building the launch-time automation tool has helped us to smoothen the development experience while making our own process much more efficient. This also helped us get the app launch time insights early on so we can work on the optimizations if needed. The best part of this process is that these insights are available to us before releasing the app! We are able to provide the best user experience every time for every user, which helps us to gain more user adoption on Dream11!

Want to help us solve our current challenges with faster, easier and better solutions? Join us at Dream11 by applying here!

Don’t let that consistent launch time slip away! was originally published in Dream11 Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.