In this article, we will provide a guided walkthrough of the Viewstamped Replication Revisited (VR) protocol. VR is a foundational implementation of State Machine Replication (SMR), and its core purpose is to enforce Total Order — the guarantee that every replica applies the exact same commands in the exact same sequence, ensuring they always converge to the same, correct state.

These concepts — SMR and Total Order — can seem overwhelming, but the good news is the underlying logic is elegant. We’ll walk through it step-by-step. At its core, VR is simply a clever set of rules that ensures every server in a cluster agrees on the exact same sequence of events, even when things go spectacularly wrong.

My own deep dive into this topic began during one of our Backend Chapter Meetings here at Thndr. A discussion about high-performance database we use — TigerBeetle — led to the mention of VR as a foundational protocol to replicate data between replicas. VR offers a unique and elegant approach to SMR that, once understood, can feel surprisingly simpler than other protocols.

The core value of this article is to provide a clear, step-by-step explanation of the protocol’s most complex mechanisms. We will focus on the engineering rationale — answering the critical “why” questions behind the design choices. As a final resource, I will share a proof-of-concept implementation to make the abstract concepts concrete.

Our Roadmap: What We’ll Cover (and What We Won’t)

The VR paper is created by Barbara Liskov and James Cowling, is incredibly detailed. To keep this article focused and practical, we will concentrate on the three core sub-protocols that define the system’s behavior:

• The Problem: Why we need replication and what we are trying to solve.
• The Architecture: The components of a VR cluster.
• Normal Operations: How the system handles requests when healthy.
• View Change: How the system handles a primary server failure.
• Recovery: How a failed replica safely rejoins the cluster.

To make sure we can cover those topics well, there are some advanced parts of the paper we’ll be skipping: things like efficient recovery, witness replicas, dynamic reconfigurations, and the state transfer mechanism. The paper is a treasure trove, and I encourage you to read it for all the details!

The Problem: Why we need replication and what we are trying to solve

Let’s start with the ideal: a single server managing a critical hashmap (your application’s state). Every command is processed sequentially. In this single-server world, Strong Consistency is guaranteed by design. Every client sees the same, correct state.

But this system is a single point of failure. If the server crashes, the state is lost.

To solve this, we introduce replication. We add two replicas, turning our single server into a cluster. The primary processes a command and sends it to the replicas.

Now, consider this sequence of events:

1. Primary processes Command A (SET ‘key’ ‘value_A’).
2. Primary sends Command A to Replica 1 and Replica 2.
3. Primary processes Command B (SET ‘key’ ‘value_B’).
4. Primary sends Command B to Replica 1, but crashes before sending it to Replica 2.

At this moment, the cluster is split:

• Replica 1 has the state after A and B (final value: value_B).
• Replica 2 has the state after only A (final value: value_A).

A terrifying question emerges: Which replica do you promote?

• If you promote Replica 2, you have silently erased a valid, committed command (Command B). The system’s history is now incorrect.
• If you promote Replica 1, you have the most up-to-date data, but how can it prove its version of history is the correct one to the rest of the world?

This is the problem: Simple replication solves fault tolerance but destroys consistency.

The solution is State Machine Replication (SMR). SMR is the discipline that forces every replica to apply the exact same commands in the exact same order — a guarantee called Total Order. VR is the rigorous protocol that enforces this Total Order, ensuring that even when a primary fails, the remaining replicas can agree on one, consistent version of the truth, and the system’s state remains accurate.

The Architecture: The components of a VR cluster

We’ve established that State Machine Replication (SMR) is the goal: forcing every replica to process the same commands in the same order. Now, let’s look at how VR is architected to achieve this.

Before we look at the players, we need to address two fundamental design questions that this distributed system must answer.

Design Question 1: How do we order events?

The most intuitive answer is to use timestamps. Every server could just timestamp the requests it receives. But what happens if the clocks on the servers aren’t perfectly synchronized? A tiny time drift — a few milliseconds here, a few there — could cause one server to order event A before B, while another orders B before A. This is why timestamps are unreliable for achieving the perfect Total Order we need. VR, therefore, needs a more robust, internal way to number and order events, which we will see in the next section.

Design Question 2: How many servers do we need?

To be fault-tolerant, we obviously need more than one. So, why not just two? Imagine we have two replicas. If they lose network connection to each other, they both continue running, thinking the other one has crashed. This is a classic split-brain scenario, and there’s no way to break the tie.

This is why VR and all robust replication protocols rely on a quorum, or a majority vote. To be able to tolerate f number of servers failing at the same time, you need a cluster of 2f + 1 servers. This formula is the bedrock of fault-tolerant design because it guarantees that a majority (f + 1) can always be reached, preventing a split-brain scenario.

With those fundamentals in mind, let’s break down the components of a VR cluster.

The Replicas (The Servers)

These are the heart of the system. Each replica is a server that runs two distinct layers of code:

• Service Code: This is the lower layer where your application logic lives — the “state machine” (our hashmap) we’re trying to keep synchronized.
• VR Code: This is the upper layer running the protocol itself. Its only job is to work with the other replicas to agree on the total order of operations before handing them off to the Service Code to be executed.

At any given time, one of these replicas will be the Primary. The Primary has the special job of coordinating requests.

The Clients

These are the applications that need to make a request to our replicated service. The client-side also has two parts:

• User Code: The top-level code trying to get something done (e.g., SET ‘key’ ‘value’).
• VR Proxy: This is a smart client library that sits between the User Code and the network. Its job is to hide all the complexity: it finds the current Primary, sends the request, and retries automatically if the Primary fails.

A Quick Note on Communication

The protocol assumes an asynchronous, message-passing model. This is crucial for a fault-tolerant system, as it means components send messages out and can continue doing other work, rather than being stuck waiting for a response from a server that might have just crashed.

Normal Operations: How the system handles requests when healthy

The “Happy Path” is where VR enforces Total Order. Every client request must pass through the Primary.

To do its job, every replica in the cluster maintains a set of state variables. Think of this as its internal dashboard for tracking everything that’s going on.

• Cluster Configuration: A simple list of all the other replicas in the cluster (their hosts and ports) so it knows who to talk to.
• Replica Number: Each replica has a unique ID, usually from 0 to N-1, where N is the total number of replicas.
• View Number: This number identifies the current leadership period, or “view”. It starts at 0 and increments every time the cluster initiates a View Change to select a new primary. This allows replicas to easily ignore messages sent by an old, deposed primary from a previous view.
• Status: The replica’s current mode of operation. It can be normal, view-change, or recovering.
• Log: The most critical piece of state. This is an ordered list of all the operations the replica has agreed to. The goal of the entire protocol is to make sure this log is identical on all correct replicas.
• Op-number: The number of the latest operation the replica has added to its log. This number always increases.
• Commit-number: The number of the latest operation that the replica knows is safely committed by the cluster and has been executed by its service code. The op-number is always greater than or equal to the commit-number.
• Client Table: A table to keep track of the most recent request processed for each client, preventing duplicate operation

With that in mind, let’s follow a single client request from start to finish.

1. Client Sends Request to Primary. The Primary receives the Request (containing the operation, client ID, and increasing request number) and immediately consults its Client Table. If the request is a retry, the Primary simply resends the previous Reply. This ensures the operation is executed only once.
2. Primary Proposes the Operation. The Primary assigns the request the next available Op-number, adds it to its own Log, and updates the Client Table. This action serializes the request, enforcing Total Order.
3. Primary Replicates the Proposal. The Primary broadcasts a large Prepare message (containing the View Number, client message, Op-number, and Commit-number) to all Backups. Backups are strict: they will only accept the operation if they have all preceding entries in their log, enforcing the total order.
4. Quorum Agreement is Reached. Backups add the operation to their log and send a small PrepareOk back to the Primary. The Primary waits for f PrepareOk messages from different Backups. Once the quorum is reached, the operation is officially committed and guaranteed to survive any single server failure.
5. Execution and Commit Notification. The Primary executes the operation, increments its Commit-number, and sends a Reply to the client. Backups learn about the commit when the Primary “piggybacks” the new Commit-number onto the next Prepare message, allowing them to safely execute the operation and keep their state identical.

View Change: How the system handles a primary server failure

The true test of any replication protocol is how it handles failure. When the Primary server crashes or becomes isolated, the cluster must heal itself automatically. This is the job of the View Change sub-protocol.

A “view” is a leadership period identified by a monotonically increasing number. The View Change is the formal, multi-step process the cluster uses to transition to a new view, establish a new Primary, and — most critically — agree on the single, definitive version of the log before starting.

1. StartViewChange: When a Backup’s timer expires, it broadcasts a StartViewChange message with a new, higher view number. It then waits until it receives f matching messages from other replicas before proceeding.
2. DoViewChange: Once a quorum agrees a change is needed, replicas send a DoViewChange message to the new, deterministically chosen Primary (e.g., new_view_number % Number of replicas). This message contains the replica’s entire state, including its log.
3. StartView: Declaring the New Truth. The new Primary waits for a quorum (f + 1) of DoViewChange messages. It then applies a strict rule to select the most up-to-date log (the one with the largest last_normal_view_number, with ties broken by the largest op_number). It then broadcasts a StartView message containing this definitive state.
4. The Cluster Gets in Line. When other replicas receive the StartView message, they adopt it as the absolute truth, replacing their log and state. The new Primary begins accepting client requests, while the Backups execute any newly committed operations to catch up. The cluster is now healed and ready for normal operations.

Since the new Primary is chosen deterministically (new_view % Number of replicas), why can’t we just switch to it immediately? The answer is that the deterministically chosen Primary might not have the most up-to-date log. It could have been partitioned or slow, and allowing it to become the primary without first learning the true state of the cluster would lead to data loss.

Also What if the new Primary fails? If the new Primary crashes during the view change, the process will time out, and the replicas will simply start another view change for an even higher view-number, choosing the next replica in line. (depends on the number faults f a cluster can handle).

Some messages like DoViewChange and StartView include the entire log, but this would be inefficient in practice. Real-world implementations optimize by sending only the suffix of the log—the most recent entries likely to differ between replicas. If the new Primary needs more context, it can request missing log segments from other replicas. Meanwhile, older log prefixes are persisted to disk, although the protocol itself does not rely on disk for correctness.

Recovery: How a failed Replica safely rejoins the cluster

What happens when a replica crashes, goes offline, and then reboots? It wakes up with amnesia — completely out of the loop, unaware of the current view, the Primary, or any operations it has missed. Before it can participate again, it must safely catch up. This is the job of the Recovery sub-protocol.

(A quick practical note: while the paper describes an in-memory protocol, any real-world system would persist the log and state to disk to survive a full reboot.)

Here is the process for a replica to rejoin the cluster:

1. The Replica Asks for Help. The newly rebooted replica sets its status to recovering and broadcasts a Recovery message to all other replicas. This message contains a unique nonce (a number for this specific recovery attempt) to ensure it only processes fresh responses.
2. Healthy Replicas Respond. A healthy replica (one in normal status) sends back a RecoveryResponse containing a snapshot of its current state (log, view number, etc.). This prevents a replica that is itself in the middle of a chaotic view change from providing confusing or incorrect information.
3. The Replica Rebuilds its State. The recovering replica waits until it has received a quorum (f + 1) of RecoveryResponse messages. It then adopts the state from the Primary’s response within that quorum, as the Primary is the definitive source of truth for the current view. The replica’s state is now perfectly consistent, and it changes its status back to normal, ready to participate in the cluster.

Why does the recovering replica need to hear from a quorum? Why not just trust the first response? The answer is about trust and correctness. A single replica might be partitioned or lagging behind. By waiting for a quorum, the recovering replica guarantees it is receiving the correct, committed state of the majority, not the potentially stale state of an isolated server.

What is the purpose of the nonce? The nonce prevents confusion from delayed network messages. It acts like a unique ticket for each recovery attempt. If the replica times out and starts a new recovery, the new nonce ensures it will ignore any late-arriving responses that belong to the old, failed attempt.

Conclusion: The Price of Agreement and a Gift

From the happy path of normal operations to the controlled chaos of a view change and recovery, we’ve walked through the core logic of Viewstamped Replication Revisited. We’ve seen how it uses quorums, view numbers, and a strict set of rules to enforce Total Order and keep a cluster of servers in perfect sync.

The Price of Agreement

Before we finish, there’s one important, practical point to make. The level of safety and agreement this protocol provides is incredibly powerful, but it comes at a cost. The multiple rounds of messaging and waiting for a quorum mean it’s inherently slower than just writing to a single server.

You should only pay this price when the cost of being wrong is very high. For many applications — like analytics or logging — it’s perfectly fine for data to be a little out of sync. Always choose the right tool for the job; this level of rigorous agreement isn’t needed everywhere.

From Theory to Code: A Gift for Making It to the End

Reading the paper is one thing, but the real “aha!” moments come when you try to translate the logic into code. As a thank you for following along on this journey, I’ve cleaned up my proof-of-concept implementation and put it on GitHub.

(A quick and very important disclaimer: this is a learning project. It is not production-ready!)

It’s meant to be a companion to the paper, helping to make the abstract concepts a bit more concrete. I hope this guide has helped demystify the protocol and that the code can help you on your own learning path.

You can find the full project here:

GitHub - MinaRaafat62/vsr-kv

Viewstamped Replication Revisited: A Step-by-Step Guide was originally published in Thndr on Medium, where people are continuing the conversation by highlighting and responding to this story.

Thndr has always been an investment platform, designed to make investing more accessible and intuitive for everyone. From the start, we focused on simplifying the investment journey by offering a user-friendly, interactive and educational experience that removes the complexities traditionally associated with financial markets, by helping people invest in different types of securities.

More than a year ago, we observed a recurring behavior among our users — they were using Thndr not just for investing but also as a place to “set their money aside” on their Thndr Wallet until they were ready to invest. Many referred to this habit as “keeping money in a drawer”: a safe space where they could park their funds before making their next financial move.

This insight reshaped our perspective on how users interact with Thndr. It became clear that Thndr wasn’t just an investment platform — it was also a de facto savings tool. But we knew this “drawer” experience could be more purposeful, intuitive, and visually appealing. Instead of just being a passive holding space, what if it could actively help users manage and grow their money?

That realization set us on a path to rethink and enhance this behavior, transforming the drawer from a simple waiting spot into a powerful, structured savings experience.

Research by economist Richard Thaler explores the notion of ‘mental accounting’. The findings are exciting and show that earmarking funds makes saving easier by budgeting your savings and rethinking spending patterns. We tried to mimic this behavior with our Savings Clouds, allowing users to organise their funds into dedicated buckets — emergencies, a new car, summer trip — keeping funds liquid while preserving a mutual fund core.

Why a mutual fund core? At Thndr, this decision was driven by several key reasons. As an investment-first platform, we’re committed to reshaping the way people think about saving and investing. By integrating a mutual fund into the Clouds experience:

• We’re making funds more approachable and effective for users looking to grow their savings with confidence.
• Ensuring regulatory greenlight since mutual funds are well-established, regulated investment vehicles that offer transparency and security, operating within our license, making them a natural fit for Clouds.

Savings Clouds go beyond just convenience — they’re designed to deepen user engagement, boost retention, and support financial well-being by making goal-based saving an integral part of Thndr. When users actively allocate funds toward their personal goals, they’re not just saving — they’re building a stronger connection with our ecosystem. By aligning their financial habits with meaningful objectives, we create a more engaging and rewarding experience that keeps them coming back.

The Product Cycle

Building products isn’t easy — if it were, we’d have fewer meetings and a lot less grey hair. Here’s what we did:

1. Identify an Issue

To build a better experience on Thndr, we’re always looking at how users interact with the platform. One key focus area is understanding withdrawals — why users move their money out and what would make them park more of it on Thndr, and for longer.

By gathering data and speaking with our users, we’ve been exploring ways to encourage more deposits and make Thndr a place where people feel confident leaving their money. Through this process, we’ve realized there’s plenty of room to improve.

Our withdrawal survey revealed that 50% of users with decreasing monthly deposits leave their withdrawn funds as idle cash outside of Thndr, indicating a missed opportunity to retain these balances within the Thndr Wallet.

2. Empathize

The survey helped us spot a gap, but it could only tell us so much. To really understand our users, we had to speak to them.

After conducting user interviews and calls, we found that users withdrawing had described Thndr’s investment wallet as a ‘drawer’ — a temporary place to hold funds while they explore new opportunities, whether for investments within Thndr or savings elsewhere. They also expressed a preference for not using these funds for daily expenses.

• 20% of users referred to Thndr as a ‘savings drawer’
• 100% of users expressed that they were waiting for an investment opportunity
• 50% of users expressed disagreement with current saving alternatives

3. Define

Time to brainstorm! With the problem clearly outlined, we shifted our focus to finding the solution (the How). We needed to address and solve the following challenges for our users:

• How to benefit from keeping idle cash on Thndr
• Have easy access to those funds, ready for any investment opportunity

We believed that offering a high-liquidity savings experience would help retain users. Psychologically, it aligned with behaviors they were already familiar with. We wanted to reward them for keeping idle cash on Thndr.

Economically, this approach leveraged the concept of opportunity cost for the user — rather than keeping idle cash in low yield savings or withdrawing it prematurely, users could maximize their returns while maintaining liquidity. By offering a high-liquidity savings option, we aimed to reduce the friction between saving and investing, increasing the velocity of money within Thndr’s ecosystem. This not only incentivizes users to retain funds on the platform but also reinforces the principles of capital efficiency creating a more sustainable financial habit.

Our goal? Making saving second nature.

4. Ideate

With our creative hats on, we began envisioning what this would like on Thndr. We knew the direction we wanted to take, but we needed to figure out how to make it fit. As a start, we identified key user needs and considerations:

• Frictionless Experience — Saving should be simple, not overwhelming. We wanted to design a seamless flow for users to allocate and track their savings effortlessly.
• Clear Separation Without Restriction — Users shouldn’t have to choose between saving and investing. We need to allow them to grow savings alongside investments.

We wanted to build something that felt both fresh and familiar — beyond investments, but still unmistakably Thndr.

5. Build & Iterate

This led to the first version of Savings Clouds — just called the Savings Wallet. (Back then, the only clouds we knew of belonged to Google and Apple).

We built a simple wallet interface showing user balance — a basic MVP. Our hooks:

• Steady returns on idle cash
• Instant transfers to their investment wallet for new investment opportunities

Sounds great, right? Well, turns out… it wasn’t.

Most users weren’t hooked.

After listening to our users and gathering feedback, we realized there was plenty of room to improve. Here’s what we learned from our MVP:

• Psychologically, users saw it differently from an investment — goal achieved, no need to engage further.
• It lacked emotional attachment. There was nothing special about the experience — it felt too basic. (Learning: MVPs don’t have to be boring).
• Users wanted to track progress and see growth, which we didn’t offer.

We had built an average product.

I was then reminded of a quote I previously heard:

“Your feature is an average of its components, not its sum.” [Spotify: a Product Story]

That hit home. We had good individual hooks, but put together, the overall experience was just… okay. And okay wasn’t enough.

So, we went back to the drawing board:

• We gave users progress visibility of their expected savings growth and actual lifetime gains. This reinforces habits and keeps users committed.
• We integrated goal-setting, helping users take control of their savings and track progress.
• We added categorization, allowing users to split savings into buckets and work toward specific goals.
• We redesigned the interface to make it fun, interactive, and engaging.
• We revamped onboarding, shifting from a bottom sheet in our MVP to a story-like experience that excited users and clearly explained the hooks.

6. Deliver

And so …drum roll… Savings Clouds were born! (With a name like Thndr, adding Clouds was the natural next step).

A feature that not only keeps users’ savings intact but makes the experience rewarding, engaging, and an integral part of Thndr’s ecosystem.

For the main screen, the MVP solely emphasized balance visualization, which didn’t add much value to users. To enhance clarity and engagement, we upgraded the look and feel, and highlighted key metrics through placeholder cards on the main screen:

• Currently Earning: An automatically updating percentage reflecting daily fund performance - annualized based on the prior 30 days.
• Lifetime Gains: A progress tracker reinforcing a sense of accomplishment.
• Clouds List: A clear view of savings across different categories.
• Progress Bar: A visual motivator, encouraging users to add more as they approach their goals.

The Savings Wallet top-up experience was simple and straightforward but this led to low user attachment. Users saw it as just another money movement, lacking a sense of achievement or celebration. The revamped Cloud creation flow fostered a sense of ownership, making users feel more engaged in the process, which led to a higher user return rate.

The onboarding experience has evolved significantly. Initially, the MVP used a simple bottom sheet with bullet points, but user feedback revealed that most ignored it. To drive engagement, we transitioned to a story-like experience, making it more interactive and exciting for users.

User Behavior Changes

So far, the results have been promising:

• Our MVP had an engagement rate (% of users interacting with the feature) of 2%, while the Clouds have an engagement rate of 10%
• Our MVP had Savings Clouds users deposit 3x higher rate compared to users that didn’t interact with feature, while the Clouds has users deposit 10x higher deposit rates
• Currently consisting of 80% of the cash of Thndr users interacting with the feature (wallet contribution)

What’s Next?

We’re committed to continuously improving Savings Clouds to deliver the best experience for our users. As we refine this feature, we’re focused on making saving not just instinctive, but completely seamless and actionable. Fintech isn’t just about providing tools — it’s about building experiences that make financial health effortless and long-lasting.

And Savings Clouds are just the start 😉

How do you design for better saving habits in fintech? Let’s discuss on LinkedIn! 🚀💬

– Read more about Savings Clouds here!
– Interested in Thndr? Sign up here!

Making Saving Second Nature: The Thinking Behind Savings Clouds was originally published in Thndr on Medium, where people are continuing the conversation by highlighting and responding to this story.

At Thndr, 2024 has been nothing but remarkable. With over 12 billion EGP in monthly trading value processed through our platform, Thndr has solidified its position as Egypt’s #1 investment platform in terms of retail value and volume. The numbers speak for themselves: an average of 1.8 million transactions per month compared to just 91K in 2022, marking one of the most exciting growth stories in the MENA region.

While such exponential growth proves our platform’s success to handle, anyone in the software engineering industry knows the complexities of scaling a system to meet surging demand. Achieving this requires building systems that are scalable, resilient, and high-performing, which demands relentless focus across various workstreams. One of the most critical areas that contributes to this success is infrastructure — the very backbone of our platform.

This article shares our journey of transitioning from a non-Kubernetes setup to a Kubernetes-based infrastructure. We’ll walk you through the challenges we faced, the trade-offs we made, and the solutions we ultimately implemented. Whether you’re an engineering leader, a systems architect, an infrastructure engineer, a DevOps engineer, or simply curious about modern infrastructure practices, we hope you find this story both insightful and inspiring. Get an inside look at the process and discover how you can leverage our learnings for your own tech transformation!

The On-Premise Infrastructure: Context and Challenges

While cloud providers have and keep making progress in improving compliance, security, and accessibility, financial services in many parts of the MENA region still face strict regulatory requirements. These often mandate that sensitive financial data be stored within the country’s borders, making on-premise infrastructure a necessity for organizations like Thndr.

At Thndr, a significant portion of our infrastructure remains on-premises to meet these regulatory demands. While this setup was sufficient during the early days, the rapid growth in user activity and transaction volume highlighted critical limitations. Upgrading our legacy on-prem infrastructure quickly became a top priority.

Our setup relied on a simple group of virtual machines (VMs), which introduced numerous challenges:

1. Limited development and CI/CD capabilities:
   - Applications were deployed as single processes without industry-standard development practices or robust CI/CD pipelines.
   - Engineers worked on manual and error-prone deployment processes, an approach associated with risks, such as deploying outdated packages.
2. Lack of automated testing:
   - There were no automated test cases to validate releases before deployment, increasing the likelihood of bugs making it to production.
   - Occasionally, deployments occurred during active market hours, compounding the risk of disruptions.
3. Scaling Challenges:
   - During periods of peak demand, the only option was to manually increase machine resources, a risky and reactive approach that could put platform stability on the line.
   - The lack of instance scaling made it impossible to dynamically provision additional instances to handle traffic spikes.
4. Operational Inefficiencies:
   - Routine maintenance tasks like upgrading machines, installing updates and security patches, or clearing logs to free up disk space, had to be performed manually.

Discovery

The previously mentioned issues highlighted the pressing need for a scalable, automated, and resilient infrastructure, laying the groundwork for the pivotal decision to transition to a Kubernetes-based solution.

Why Kubernetes?

Kubernetes is our usual go-to choice for deploying and managing containerized applications since it supports flexible, scalable, reliable, and efficient environments. It makes nearly every aspect of management and deployment easier thanks to features like:

1. Automated rollouts and rollbacks
2. Service discovery
3. Load balancing
4. Storage orchestration
5. Self-healing

When choosing a suitable Kubernetes cluster with limitations such as unavailable cloud solutions, you must work with a group of on-premises machines to create a semi-managed or unmanaged solution. To build Kubernetes on on-premises machines, there are several options available, including k3s, kubeadm, and EKS Anywhere.

Choosing the Right Kubernetes Solution: Balancing Constraints and Expertise

When setting up a Kubernetes cluster in your system, the chosen approach, tools, and version are determined by your constraints, areas of expertise, and existing toolset. At Thndr, we opted for k3s after extensive research and experimentation. K3s proved to be the optimal solution for Thndr. It required only a few on-premises machines with a lightweight Kubernetes versio

The Way to K3s

What is K3s?

K3s is developed by the Rancher team, and it is a CNCF Sandbox Project. K3s represents a streamlined, simple-to-install, deploy, and oversee iteration of standard Kubernetes (K8s). K3s is an authorized Kubernetes distribution. While K3s is an enhanced variant of Kubernetes (the original version), its fundamental operation remains unaltered.

Why K3s?

1. Even though the team had limited experience with setting up and configuring k3s, we didn’t encounter significant difficulties deploying k3s in our development and production environments.
2. Rancher reduced the complexity of K3s by removing over 3 billion lines of code from the Kubernetes source. They trimmed down non-CSI storage options, experimental features, and outdated components that were not crucial for fully implementing the Kubernetes API.
3. The single binary file is less than 100MB in size, enhancing speed and reducing resource consumption compared to K8s. Unlike Kubernetes, the master, nodes, and workers do not need to run in multiple instances to boost efficiency.
4. The memory footprint is reduced primarily by running many components inside of a single process. This eliminates significant overhead that would otherwise be duplicated for each component.
5. Certified Kubernetes distribution: Well supported and maintained, expected to remain as such for the foreseeable future.
6. Easier and faster installation and deployment: K3s take seconds to minutes to install and run.

K3s’s Architecture

1. A server node: Is defined as a host running the k3s server command, with control-plane and datastore components managed by K3s.
2. An agent node: is defined as a host running the k3s agent command without any datastore or control-plane components.
3. Both servers and agents run the kubelet, container runtime, and CNI. See the Advanced Options documentation for more information on running agentless servers.
4. High-Availability: For environments where uptime of the Kubernetes control plane is critical, you can run K3s in an HA configuration. An HA K3s cluster comprises:
5. Embedded DB: An embedded etcd datastore is stored inside each server node.
6. External DB: An external datastore (such as etcd, MariaDB, etc.) is placed outside the cluster in a centralized place.

Install k3s

In that section, we will focus on the core components we have, starting from how we created the k3s and ending with how we receive the traffic.

Ansible

We rely on Ansible for configuration management to set up k3s, with special thanks to Vincent RABAH, who created the Ansible file structure for k3s setup. He is the author of many other valuable resources, so we encourage you to follow his work. Before proceeding, we recommend familiarizing yourself with the core components of Ansible through this link.

1. What is Ansible? Ansible is an open-source automation tool for configuration management, application deployment, and task automation. using simple YAML playbooks to define tasks, making infrastructure management more efficient.
2. Setting Up k3s: When installing k3s, you’ll have a group of machines. You can easily designate which machines act as masters and which as agents by configuring the inventory.yml file.
3. Default Database Setup: By default, K3s uses an embedded DB. If you prefer using an external database, this can be done by adding the “-datastore-endpoint” as an extra argument and setting the “use_external_database” flag to true in the inventory.yml file.
4. Upgrading k3s: To upgrade k3s at any time, simply update the “k3s_version” in the inventory.yml file with the desired version and run the provided playbook to upgrade all nodes in the cluster.
5. Kubeconfig: After successful bringup, the kubeconfig of the cluster is copied to the control node and merged with ~/.kube/config under the “k3s-ansible” context.
6. kubectl: Start with installing kubectl, you can confirm access to your Kubernetes cluster with the following: “kubectl config use-context k3s-ansible” and then “kubectl get nodes”

Supporting LoadBalancer Services in On-Prem Kubernetes Setups

On-premises Kubernetes environments lack the cloud-native support for LoadBalancer-type services found in platforms like AWS or GCP. Tools like Cilium, MetalLB, and ServiceLB fill this gap, each with a different approach.

• ServiceLB: Built into K3s, ServiceLB exposes services via node IPs. It’s simple but lacks true load-balanced Virtual IP (VIP) support, limiting flexibility for complex setups.
• MetalLB: Designed for bare-metal clusters, MetalLB uses either:
• L2 Mode: Broadcasts the LoadBalancer IP over the local subnet, offering simplicity but slower failover.
• BGP Mode: Advertises LoadBalancer IPs as routes to the network, enabling dynamic routing and scalability but requiring more configuration.
• Cilium: An eBPF-based tool focused on networking and security. It provides high-performance load balancing and direct kernel-level routing, though it may be overkill for simple setups.

Each tool offers unique trade-offs. ServiceLB is suited for basic needs, MetalLB balances simplicity and flexibility, and Cilium delivers advanced performance for modern networks.

Managing Storage in On-Prem Kubernetes with CSI Solutions

Kubernetes, originally designed for stateless applications, now supports StatefulSets to manage stateful workloads with persistent volumes. On cloud platforms, storage is dynamically managed using drivers like AWS EBS, GCE Persistent Disk, or EFS CSI Driver, enabling seamless scaling and reliability. However, in on-prem environments, storage management is more complex due to the lack of native, scalable storage options.

Container Storage Interface (CSI) solutions address this gap by enabling dynamic provisioning, replication, and failover for persistent volumes in on-prem Kubernetes clusters. Key CSI tools include:

• Longhorn: Provides lightweight containerized storage by replicating data synchronously across nodes for high availability. It ensures resilience by automatically switching workloads to healthy replicas during node or disk failures.
• Rook: Integrates with distributed storage systems like Ceph, offering advanced features like object, block, and file storage. It is ideal for clusters requiring flexible storage types and scalability.
• OpenEBS: Focuses on per-application storage, allowing developers to create lightweight volumes with specific requirements. It’s suitable for workloads needing fine-grained control over storage.

Each CSI solution varies in complexity and capabilities, from Longhorn’s simplicity to Rook’s versatility and OpenEBS’s application-centric design. These tools make on-prem storage management more reliable, scalable, and adaptable to diverse workload needs.

Unlocking New Advantages

• Modernizing Applications: Modernizing code involves restructuring and improving software to enhance performance, maintainability, and efficiency, ensuring it remains scalable and adaptable to evolving needs.
• Streamlined Continuous Integration (CI): A robust CI pipeline includes building container images, running test cases, tagging images with semantic versioning, performing security checks using tools like vulnerability scanners, and pushing validated images to a Container Registry (e.g., Docker Hub, GitLab, or JFrog).
• Automated Continuous Deployment (CD): GitOps tools streamline deployments by syncing Kubernetes configurations directly from a Git repository. These tools support automation, scheduling, and tracking changes, enabling teams to deploy updates reliably and efficiently.
• Resilient Service Mesh and Ingress Gateway: Adopting a service mesh simplifies microservice management by enabling features like rate limiting, circuit breaking, and advanced observability through request tracing and performance monitoring. By injecting a sidecar container into each service, the service mesh provides granular insights and control over inter-service communication.
• Unified Secrets Management: Managing sensitive data has improved significantly with tools that integrate with Kubernetes, ensuring secure and stable handling of credentials. Solutions like Consul, Azure Key Vault, and others offer centralized management, reducing the risks of exposure.
• Dynamic Auto Scaling: Kubernetes supports Horizontal Pod Autoscaler (HPA) for scaling applications based on CPU and memory usage. For additional flexibility, tools like event-driven scaling frameworks can scale workloads dynamically based on custom metrics or schedules, ensuring readiness for traffic spikes, such as those occurring during peak business hours.
• Comprehensive Monitoring and Observability: Monitoring tools enable automated tracking of CPU, memory, storage, latency, and errors. Alerts can be sent to collaboration platforms like Slack or escalation tools, helping teams address issues proactively. Popular tools include Prometheus, Grafana, and log aggregation solutions.
• Proactive Incident Alerting: Incident alerting tools ensure timely responses by escalating alerts through predefined workflows. For instance, alerts may initially notify via messaging platforms and, if unacknowledged, escalate to on-call engineers and managers, ensuring issues are addressed promptly. Tools such as OpsGenie and Spike.sh support this process.
• Efficient Application Package Management: Managing configurations separately from application code ensures consistency across environments (e.g., staging vs. production). Tools like Helm enable templating and modular configuration management, simplifying the deployment of applications across clusters with environment-specific details.

Final Thoughts

The journey from a simple on-premise setup to a robust Kubernetes-based infrastructure has been one of the most impactful and prioritized projects at Thndr. While the transition presented its fair share of challenges, it has empowered us to scale effectively, deliver a more reliable and seamless experience for our users, and adhere to the regulations — all while positioning the platform for long-term growth.

As Thndr continues to expand in Egypt and the broader MENA region, we remain dedicated to evolving our infrastructure. Our goal is not just to meet the demands of today but to anticipate and prepare for the opportunities of tomorrow. This commitment is fueled by our mission to democratize investing, leveraging technology to provide a simple, barrier-free investing experience that is truly user-centric.

Introducing Self-Managed Kubernetes at Thndr: A path to Modern Deployment was originally published in Thndr on Medium, where people are continuing the conversation by highlighting and responding to this story.

Thndr App is a digital investment platform with the purpose to simplify investing. Thndr facilitates access to the right tools and resources that can empower investors with the means to achieve financial freedom.

At Thndr, we have growing number of users and with that we constantly enhance our schema configs and it’s hard to keep track of changes happening within metadata. Since it’s a financial application, we have a core responsibility to protect sensitive customer data; hence, underscoring the significance of data governance framework.

When we were looking for solutions, we tried different tools and we came across OpenMetadata, which is an open-source tool to get complete insights regarding our data from all sources, keep track of changes happening within the schema and tables and for detecting PII sensitive and non-sensitive PII columns, building glossary, and for having data quality alerts. Thus, it’s an all in one tool for data cataloging, data governance, data quality, data lineage, and advance monitoring of data using custom metrics.

When the data increases, it’s difficult to manually keep track of sensitive columns within data, data being fetched and updated on time, quality of data, and changes happening within the data present in data lake or data warehouse. Thus, it’s the responsibility of the data engineering teams to set and communicate a responsible data governance model and data cataloging framework. This helps the data teams to be aware about the changes happening within their schema, tables, stored procedures, and views.

OpenMetadata Features

• Metadata Versioning: OpenMetadata captures the changes within the schema along with changes in the volume of the data. It tracks all the changes happening within the schemas. For instance, if there is a new column added into the table, it will detect schema changes.
• Data Profiling: Openmetadata has an embedded data profiler which compute metrics on tables, such as unique records, null proportion, row count, data trends, etc.
• Data Lineage: Supports a comprehensive lineage for all data assets by capturing the relation between entities. Traces the path of data across tables, pipelines, and dashboards.
• Central Repository for Metadata: A central store to integrate metadata from different sources in the data ecosystem. Uses standardized Schemas and APIs. Brings together the power of multiple applications — data discovery, quality, observability, profiling, user collaboration, lineage, and much more. It fetches DDLs, stored procedures, views, tables, and column details for the configured schemas.
• Column Level Lineage: In OpenMetadata, we primarily compute column-level lineage through SQL query analysis. Lineage information is available from various sources, such as ETL pipelines, data warehouse, databases, dashboards, and so on.

Getting Started with OpenMetada:

There are two ways to deploy OpenMetadata. The first method is to deploy on Kubernetes using helm charts and the second method is to deploy using docker image on virtual machines using docker swarm. OpenMetadata uses different types of internal storage (PostgreSQL, MySQL) and for indexing and searching, it uses Elastic search. It is possible to deploy database and elastic search on the same machine in dockerfile or use AWS based managed RDS and Elastic search. The good thing about OpenMetadata is that it also provides google authentication and role based access control.

Metadata Ingestions with OpenMetadata

OpenMetadata provide multiple connectors through which it can fetch metadata from and run multiple types of ingestions on it. Some of the connectors include postgres, Mysql, superset, Glue, S3, Redshift, etc.

In order to create catalog in OpenMetadata, create connection with the database by clicking on Settings and then select Databases and click on Add New Service. Select service as redshift or the service that is needed for ingestion. Enter the credentials of redshift and click on save, it will test the connection initially.

Openmetadata provide five types of ingestion based on the need. For data cataloging, click on settings->databases and select the database and click on ingestions. Select Add Metadata Ingestion and choose the ingestion workflow. Initially, start with metadata ingestion.

After choosing metadata ingestion, specify database name, schema name, and table. Regex pattern can also be provided. This ingestion will be scheduled and it will record all changes happening within the metadata.

Run the ingestions and if it fails, the error can seen from the logs if logs were enabled during ingestion.

Data Cataloging with OpenMetadata

Data Catalog is built on top of all the metadata from the databases or data warehouses to make it easy for the users to search by schema name, table name, column name, and filter them based on multiple options. This will help them to flag sensitive columns. OpenMetadata offers inbuilt data cataloging feature with many different datasets and tables, each with varying access levels for different users, a data catalog provides a single, unified user experience for quickly discovering these datasets.

Once the metadata and profiler ingestions are completed, the metadata, DDL, views, and stored procedures will be available and can be explored through data catalog. In the search bar, one can discover data through table name, schema name, column name, etc. It can search from multiple datasets at a time. It also has few filters like if you want to filter on the basis of database, domain, tier, service, schema, column, or table type (regular, view).

Data Profiling with OpenMetadata

One of the best feature of OpenMetadata is that it provides all the information regarding the tables itself. It not only fetches the metadata for the schema tables but also compute metrics

Profiling data and communicating quality across the organization is core to OpenMetadata. While numerous tools exist, they are often isolated and require users to navigate multiple interfaces. In OpenMetadata, these tests and data profiles are displayed alongside your assets (tables, views) and allow you to get a 360-degree view of your data.

When the profiler is completed, it shows column profile for each table with inbuilt metrics like Null percentage, Unique percentage, distinct percentage, row count, value count, size of data, etc.

Data Governance with OpenMetadata

Data governance plays an important role in safe guarding confidential data especially PII columns. This is important so that the access to this data can be prevented across the organization and access should be granted to only relevant people. Manually identifying PII columns is time consuming and can lead to errors or leaving behind certain data; thus, OpenMetadata uses machine learning to auto classify PII columns into sensitive and non-sensitive. It uses different policies like GDPR to flag these columns.

id_number is marked as sensitive using PII auto tagging during profiler ingestion. Likewise, it classifies email, phone number, address, account details, etc. as PII Sensitive. On the other hand, it marks first name, last name, etc as PII Non-sensitive.

Data Quality with OpenMetadata

Data is received in raw format from different applications/ operational databases in production and variety of sources; hence, it may have missing values, whitespaces, invalid characters, outliers, incorrect data types, duplicates, etc., which can be problematic for Data Analyst and Data Scientist. Noisy data can ruin the ML Pipelines and skew the analytical results and won’t help in drawing conclusion from the data. Thus, with OpenMetadata, we can detect different types of data quality issues like duplicates, null count, and any custom SQL query to check any constraint like missing records, etc. If the test case fails, an alert can be configured on slack, email, teams, etc. Based on these alert, transformations can be done on the data.

Data Lineage with OpenMetadata

OpenMetadata offers a detailed solution for tracking and visualizing data lineage, which is crucial for understanding data transformations and usage within an organization. It supports lineage for data warehouses, data lakes, databases on both table and column-level traceability. The lineage can be added using an ingestion and once the profiler is completed, we get a visualization representing the connectivity and relationship between the data. This is helpful when you want to track origin of your data.

And… To Conclude!

I hope the article provides detailed overview of the latest features of OpenMetadata and how important it is. The latest version has a lot of improvements and they also offer APIs. The best thing is that it’s free and open source.

Data Cataloging and Governance at Thndr was originally published in Thndr on Medium, where people are continuing the conversation by highlighting and responding to this story.

Thndr App is a digital investment platform that is designed to simplify investing. Thndr facilitates access to the right tools and resources that can empower investors with the means to achieve financial freedom.

Discover a financial revolution with Thndr App’s Investment Guide — crafted to simplify the staggering task of starting your investment journey. Triggered by user inquiries on social media, this feature resulted in an impressive ≈8 percentage point increase in funded accounts to signups. Tailored for all investors, it enhances user experience, guiding decisions aligned with financial goals and risk tolerance. Join us to unveil the logic, explore diverse personalities, and gain insights from usability testing sessions.

The backstory of our game-changing Investment Guide

We all know it is overwhelming to start investing and find a good investment to start your journey with. So, why not help investors discover their investor personality and give them recommendations to improve their path to financial freedom?

Whether a beginner or an experienced investor, the Investment Guide feature helps you make informed decisions aligned with your financial goals and risk tolerance.

What was the trigger?

We received many CX inquiries and users were also asking on our social media platforms what to invest in. And, a lot of them weren’t familiar with the different investment products on Thndr.

A deep dive into tailoring our guide for maximized impact

We wanted to create a fun, short, and effective guide to have accurate outcomes. Hence, we focused on 4 main aspects that will determine an investor’s personality:

1. Objective: Determine a clear and specific investing goal.
2. Risk: Measure the tolerance for uncertain or adverse actions in a given situation.
3. Duration: Analyze the investing time frame.
4. Passiveness: Compare activity vs proactivity.

After pinpointing these 4 aspects we created 4 different personalities for all the possible results. Here they are:

1. Guardian: You know your money’s worth and want to guard your hard-earned treasure.
2. Maverick: You prefer to take control of your own decisions — and you know very well that with high risk comes an even higher reward.
3. Portfolio Pro: You’re open to some risk, but prefer to leave it up to the pros to invest on your behalf.
4. Safe-Player: Slow and steady wins the race; you’re not about taking chances.

Cracking the Code: Unveiling the Logic Behind Our Results!

Here’s a diagram illustrating various scenarios and their corresponding outcomes.

Are you curious about exploring different personalities? — we’ve got you covered

We provided a flow where you can view all personalities after you finish the guide. Browsing through other personalities and learning about them might motivate investors to explore more opportunities. We also allowed the users to redo the guide if they wanted to.

Usability testing sessions for an extra layer of confidence

We tested with 5 users to understand their behaviors and to make sure the flow was clear and easy to digest.

The Moment of Truth: Examining how well this feature has delivered

An increase of ≈ 8% in the conversion of funded accounts after releasing this feature.

In conclusion, introducing the Investment Guide on Thndr App has proven to be a pivotal step towards enhancing the user experience and encouraging informed investment decisions. Recognizing the overwhelming nature of starting an investment journey, the Investment Guide assists both beginners and experienced investors in discovering their investor personality and aligning their choices with financial goals and risk tolerance.

And… That’s a wrap!

I hope you found this case study useful and informative. Feel free to contact us on any social media platform with any questions. We would be more than happy to have a chat with you.

The Product Design Story behind a boost in Business-funded Users during Q3–2023 was originally published in Thndr on Medium, where people are continuing the conversation by highlighting and responding to this story.

Have you ever found yourself in one of these team retrospectives, where everyone tried to figure out why everything went wrong? missed iteration goal for the second time in row, last-minute changes, unstable releases, and a hell of ad-hoc requests. Frustration and confusion were all over the room. I believe you have, and I also think you were fortunate to be in a place where positive vibes were the theme of a retrospective, when your team felt productive and happy to deliver impactful stuff. A lot of aspects affect team efficiency and morale, one of which is product managers.

Over the past nine years, I’ve been working almost every day with product managers and owners on numerous products that are meant to help people, give them the right value, and make their lives easier. As an engineer, I always find it challenging for product managers to keep the right level of dedication and focus on many areas like product discovery, stakeholder management, roadmapping, setting product vision and strategies, release and GTM planning, competitor analysis, and, of course, enabling and empowering their engineering teams, the real product builders!

I’ve witnessed a lot of missed opportunities for teams to produce timely and quality outcomes and results for the products they worked on, despite the fact that these teams had plenty of great talents with solid experience under their belts. They had the same goals and ambitions, but what kept them from reaching their potential? I would say there were different reasons that led to this, but at the top of the list was the inability to devote enough time and attention towards crafting solid processes and building teams with high momentum and autonomy.

In fact, one of a product manager’s main responsibilities is to work closely with the engineering teams. Effectively doing this contributes to the PM’s success in the organization. Being unable to handle it might introduce some complications, like frequent missed iteration goals or being such a bottleneck that the team always needs them there to make a decision. In this article, I will talk about these issues and many more. I’ll also list 10 key characteristics that I’ve noticed in some of the fantastic product managers I’ve worked with as a team member and leader of many engineering teams. These qualities have helped their teams unleash great acts of efficiency and deliver results that immediately impacted their products.

Note: This article is packed with some cool stuff taken from the fantastic Dilbert Project. I hope you like it.

1. They spend a good deal of time making themselves available to their teams.

Most of us had the chance to work with those product managers who were so busy that they rarely came to our stand-ups, replied back on Slack threads, or even attended retrospective meetings. They prepare just enough for other scrum events like pre-planning, backlog refinement, and sprint planning meetings, and that’s fine. The problem is that, in many cases, it is necessary to have a responsive product manager who can give the team answers and guidance as needed during the sprint. We never assume that within two to three weeks, we’ll have a well-defined scope with clear acceptance criteria. The best product managers always make time to participate in ad-hoc calls or to respond to a thread in which the front-end and back-end engineers argue about something.

2. They ask the right “Why?” questions and bring solid problem statements to the team.

Product and engineering teams may find themselves frequently asking, “What do we need to build next?” as they have numerous boards piled high with features and requirements waiting to be picked. However, product excellence is not determined by the quantity of items delivered; it is determined by the value these items add to consumers and other stakeholders. That’s why it is essential to take a breather every so often and switch from asking “what?” questions to “why?” questions whenever possible.

Experienced product managers never present their teams with solutions but rather with problem statements. For instance, instead of being asked to implement a user provisioning feature using CSV sheets, the team working on the B2B product should instead hear about the problems that the users are having when it comes to onboarding new corporate customers. Clear problem statements let the team use their creativity, product knowledge, analytical and critical thinking skills, and other skills to solve problems and come up with good ideas. Delivering features alone won’t solve issues; providing value will. Because of this, it is a product manager’s responsibility to conduct appropriate discovery and requirement gathering in order to enhance the conversations they have with their teams. That should lead to a list of possible solutions that can be put into action to solve the problem that was already mentioned.

3. They don’t turn their products and teams into “feature factories”.

A “feature factory” is a product or company that measures output over outcomes. It’s a story point machine where the company only cares about how much is being shipped. The company celebrates the work done, and it feels like a conveyor belt going in one direction. No one is thinking about the question: is any of this really working? No one is thinking if any of this complexity is adding business value.

A “feature factory” is a place where the outputs are celebrated rather than the outcomes. Teams can’t be sure that their work really helps customers, so during sprint planning, they only focus on shipping the feature asked for by the product manager. There is no impact analysis, success or failure metrics, or even a plan for what will be built on this feature after it is released.

Occasionally, in feature-factory types of teams, feedback on the deliverables is provided without the participation of the individuals who worked on them. Customers, stakeholders, and product managers all participate in the feedback loop. A good product manager will ensure that their team is kept in the loop at all times. It’s always important to make sure that engineering teams can see the organization’s goals, the metrics used to measure how well the results are working, and the direction that each of us should be going. Whenever they see a benefit from including the engineering teams in stakeholder discussions, they make sure to do so without overburdening them.

They keep giving feedback on what the team did, celebrate when things went well, and give the team the credit they deserve. Additionally, they are eager to discuss concerns and errors the team has made in order to improve in the future.

4. They can deal with unplanned work problems.

No matter how well a team plans its activities, there is always going to be some unexpected stuff popping up in the middle of the sprint, and the product manager and the team need to find an efficient way to deal with it. Unplanned work comes in many forms, such as emergency production bugs, data corruption, a small change in a requirement that stops us from launching a new feature, or an angry customer or stakeholder that we need to do our best to keep.

As you can see, there are countless possibilities, and the product manager is the team’s first line of defense. He or she should be able to effectively communicate with the issue or request reporter, prioritize, evaluate the urgency, and consider how to push back if necessary. They should also communicate with the team about whether to incorporate the item into the current sprint or substitute it for something else. Finally, he or she should be doing so cautiously to make sure that the team’s focus and morale won’t be affected.

5. They enjoy getting into technical discussions with their teams.

Being a technical expert with hands-on experience is not one of the many skills needed for product management. The ability to read between the lines, take in the information at hand, and lead productive conversations with engineering teams is, however, indispensable. Without it, it would be hard for product managers to figure out why the team needs two sprints to show or hide buttons on a screen that has been there for four years and hasn’t been touched since then.

Product managers must work to gain a thorough understanding of the system architecture, how each component interacts with other services, and the technologies and tools the team is using. By doing this, they will have a better chance of getting involved in the team’s technical threads and knowing what system limits we need to work on or avoid until we deal with the team’s technical debt.

Understanding various software development processes and terminologies is essential. A good product manager is aware of the team’s process for releasing new features, how production bugs are fixed, and what the evil term “CI/CD” is all about.

6. They can weather stakeholders’ storms and protect their teams.

There are a lot of ways to define the word “stakeholder.” From my point of view, a stakeholder is someone who is interested in the product in different ways. This could be a customer, a shareholder, someone who has a say in the product’s direction and strategy, or anyone who will be affected directly or indirectly by changes to the product. Based on this definition, managing stakeholders has always been seen as one of the hardest jobs anyone could have, and product managers can’t avoid doing it.

Stakeholders will always want to have their requested, urgent features in the product as soon as possible. This is where product managers’ skills in negotiating, refining the scope, setting priorities, and keeping everyone in the room convinced and happy come into play. It is essential to provide the engineering team with a sufficient amount of time to discuss requirements and evaluate feasibility. After that, it is the responsibility of the product manager to vouch for the team’s decisions and estimated delivery timelines. Most of the time, product managers are dragged into conversations where they have to commit to a timeline. If they do this without making sure their teams are on the same page, it could lead to unnecessary pressure, a lot of context switching, and a lower level of quality in the end result.

7. They are data-oriented people who always speak in numbers.

When it comes to numbers, we’re talking about the right ones, because if we measure the wrong things, it’ll only make the situation worse over time.

I’ve known product managers who gauged product success based on the number of contracts won and the number of new features released each quarter. These numbers are ineffective because they indicate nothing other than a lack of product vision. Better metrics and measures should pay attention to aspects like customer retention and acquisition rates, user satisfaction and reported complaints or crashes, the impact of delivered features, etc.

It is always difficult to convince a competent engineering team that asks the right questions about features or requirements that are supported by nothing but a product manager’s sense of urgency. The team should have the freedom to contest the necessity of a feature, asking for the impact in numbers and supporting data. The product manager should be ready for these talks and have put in the time and effort to gather these numbers and data for their teams, not to mention the market and user research that should have been done before this step.

8. They create well-written documents that explain almost everything.

The ability to create excellent documentation is one of the abilities I’ve observed in every reliable product manager I’ve worked with. How well a product manager can communicate concepts, problem statements, product vision, and a variety of other things to the rest of their team will largely determine how effective they are. It’s helpful to be able to articulate ideas and document them so they can be shared with engineers, sales reps, business stakeholders, etc.

These documents could save a ton of time and effort in explaining product things and instead give the readers the chance to absorb the information and work on it asynchronously, which is more efficient, especially for engineering teams. Everyone should always look to well-written user stories and PRDs (Product Requirements Documents) as their source of truth, not just the engineering team that is working on the implementation of the feature.

9. They contribute to the growth of more product-minded, autonomous engineers.

Every problem that a product manager brings up shouldn’t always be solved by a technical solution. No matter how technical the ideas or solutions are, a team’s level of excellence is always determined by how well it can come up with good ideas and solve problems. Team members should be able to challenge problems and modify the existing system to meet requirements with minimal technical modifications. Not only should they lead the “How?” phase, but they can also suggest changes to the scope that would result in greater value. Product managers who are smart allow their teams to participate in all phases and do not just see them as executors who arrive late to the product evolution phases.

Additionally, confident product managers give the team the room and degree of autonomy required to make tactical decisions and manageable scope changes that benefit them but don’t detract from the project’s overall deliverables. By doing this, the product manager won’t become a bottleneck who needs to be consulted on even the smallest changes.

10. They create solid GTM plans for product releases.

A go-to-market plan, also known as a GTM plan, is an important step that needs to be taken in order to guarantee that the users you have in mind will be aware of the launch of a new product or feature. This plan should cover a wide range of topics, including who will use this feature, which platforms should be used, what are the prerequisites, how should we notify the users, what are the expected outcomes, how will it be managed and handled for potential issues and onboarding requests, etc. It should also contain a clear set of action items with their respective owners.

The team needs to be aware of what is expected of them during the development, release, and launch phases, so this plan needs to be open to changes as we go forward and visible to all stakeholders. This gives them confidence that they’re working to a plan and won’t be surprised by unplanned tasks like user provisioning or technical support.

Final Thoughts

It always takes time to build a strong team that performs well and whose members are confident in each other’s judgment and the decisions they take. Product managers should receive the utmost support because they are the ones who carry the flame and light the way for their engineers. In addition, it is their responsibility to ensure that the team is focusing its efforts on activities that are beneficial to them and the organization as a whole.

How do Product Managers help their Engineering Teams excel and deliver better outcomes? was originally published in Thndr on Medium, where people are continuing the conversation by highlighting and responding to this story.

Hey, it’s Redis again!

Redis is a powerful in-memory data structure store that has become increasingly popular among developers for its speed, versatility, and ease of use. With its ability to handle data structures such as strings, hashes, lists, sets, and more, Redis is a great tool for solving complex problems in real-time applications.

At Thndr we rely heavily on Redis. Our use cases vary between caching, distributed rate limiting, Pub/Sub for most of our background jobs, and even as a persistent key-value store, among others. And while at first it wasn’t an issue, now, with our growing user base, which grew by more than 400% in the last 5 months, we started seeing a substantial increase in our Redis latency.

We’re using AWS ElastiCache as a managed Redis solution. At the beginning of all of this, we had a single Redis server and database provisioned and used by all our microservices. Some of the data that lives in this database is used by multiple services, and some of it is scoped to a single service. This is the famous “common data coupling” anti-pattern, and gradually, we started seeing more and more issues with this setup, increased latency being one of them. First, we hit our instance bandwidth limit, so we went ahead and scaled our instance vertically so that AWS would give us more bandwidth. The problem was that we were wasting a lot of money as our CPU usage was less than 7% and our memory consumption didn’t exceed 22%. Nonetheless, this solution helped us become resilient for a period of time. We would still have some latency spikes in market hours that could go up to 1s+ (ideally, Redis should have a latency of 1 ms for writes and much less for reads), but we were coping.

We have a problem..

Soon after, this past December, we had our first big incident. It started with a huge spike in latency. Redis requests queue became backed up under the sufficient load, where the requests were being queued much faster than they were being processed. During the incident, Redis’ p99 latency reached the 40s mark. This affected our SLA and the user experience negatively. Our most critical services were affected, and the entire app became very slow. We saw 1+ minute latencies in some of our most important services. This was truly a wake-up call that relying on vertical scaling is not an option anymore and that it was time we addressed the root cause of the problem at hand.

The following graphs demonstrate how this had a significant impact on our latency throughout that incident.

Discovery

As mentioned before, we hit our bandwidth limit in Redis, partially because of our growing user base, in addition to some anti-patterns that we have scattered around the codebase, but more on that later.

Redis is single-threaded by nature, which means no two commands can execute in parallel. This might appear as an issue, but in most cases, network bandwidth and memory bandwidth are the bottlenecks as opposed to the CPU; thus, the performance gain from multi-threading is negligible. Since most threads will end up being blocked on I/Os, it wouldn’t justify the overhead of multi-threading, which includes significant code complexity from resource locking and thread synchronization and CPU overhead from thread creation, destruction, and context switching.

Nevertheless, when you use it to perform multiple key operations periodically on hundreds of thousands of keys, like we were doing at the time, it can take a lot of time to process and block the event loop until they finish. These expensive O(n) operations, like mset and mget that we used to run, attempt to get more than 25k values at a time. There’s also the usage of the KEYS command, which is a very expensive and blocking operation and should be avoided at all costs in production and replaced instead with SCAN.

Through monitoring tools (we mainly rely on Datadog for this), we found out that unfortunately, our most impacted service is the market service, where all of the trading and most of the magic happens.

Operation “Split and Scale”

First, we started by splitting our huge, monolith Redis database into multiple ones, one for each service. We were able to end up with a Redis database per service for most of our services, except for a few where there’s this common data coupling issue that we mentioned in the intro. Already, by doing this split, we were able to give ourselves a lot of headroom and buy ourselves some time, to focus on the other contributing factors.

Then, we started looking into how to scale the different Redis instances, and naturally, we considered Redis Cluster. Since Redis operates with a single-threaded model, vertical scaling is not really a helpful option, as it is not capable of utilizing multiple cores of the CPU to process commands. So instead, horizontal scaling seems like a more plausible solution. Redis Cluster utilizes horizontal scaling by not only adding several servers and replication the data but also distributing the data set among the nodes in the cluster (what you’re maybe familiar with as sharding) enabling the processing of requests in parallel. What makes Redis Cluster extra special, however, is its sharding algorithm; Redis Cluster does not use consistent hashing, but a different form of sharding where every key is assigned to a hash slot.

Hash slots share the concept of using hashes or composite partitioning but do not rely on the circle-based algorithm upon which consistent hashing is based. One of the drawbacks of consistent hashing is that as the system evolves, operations such as addition/removal of nodes, expiration of keys, addition of new keys, etc. cause the cluster to end up with imbalanced shards. In a way, this would have created more problems instead of solving them.

The way “Hash Slots” effectively solve this is by partitioning the key space across the different nodes in the cluster. Specifically, it is split into 16384 slots where all the keys in the key space are hashed into an integer range 0 ~ 16383, following the formula slot = CRC16(key) mod 16383. To compute what the hash slot of a given key is, we simply take the CRC16 of the key modulo 16384. Doing this makes the evolution of the cluster and its processes (such as adding or removing nodes from the cluster) much easier. If we are adding a new node, we need to move some hash slots from existing nodes to the new one. Same way, if we would like to remove a node, then we can move the hash slot served by that node to the other nodes present in the cluster. In return, this creates a much more balanced cluster, solving the issue with consistent hashing.

One thing to take into consideration though is that AWS Elasticache operates a little bit differently than Redis. So, for example, there is no sentinel mode, so sharding is always obligatory if you’re thinking of adding more nodes to scale horizontally. Also, cluster mode behaves a bit differently than how Redis normally does. Like for example there is no minimum limit of three on the number of nodes that a cluster should have as Redis does. So AWS Elasticache documentation should be consulted first before going through with it as a solution expecting it to behave just like Redis.

Another Discovery

We also looked at the library that we are using now to make calls to our Redis databases. We found that it was deprecated, and didn’t support Redis Cluster. Digging deeper into its source code, we found that it was also not that performative in terms of some multi-key commands. For example, running commands in a pipeline actually runs the commands in a for loop instead of actually pipelining the commands. We decided that we should look at other libraries that support Redis Cluster, which are hopefully much more performative.

Another area for improvement was the fact that we use Redis as a Pub/Sub mainly to schedule celery jobs. In a nutshell, using Pub/Sub with Redis Cluster is generally a bad idea. As the client can send SUBSCRIBE to any node and can also send PUBLISH to any node, the published messages will be replicated to all nodes in the cluster. This makes Redis Cluster as a Pub/Sub solution inefficient.

The Solution

The first thing we did was replace our Celery jobs with Kubernetes jobs instead across all our services to mitigate the load a bit from Redis. An added bonus was that we have much better observability on Kubernetes jobs than we did on Redis jobs anyway.

Then, as mentioned before, our market service is not only the most impacted service, but it’s also one of the most important and most used services in our ecosystem, so we decided to start from there.

We wanted to try implementing Redis Cluster there first and monitor its performance. And since this is a big move, we took it in steps, so we first looked at alternatives to our deprecated Redis library. Our criteria for choosing this library were 3 things;

• Huge support from the Redis community
• Performative operations where for example pipelining is actually pipelining the operations and not just putting them in a for loop under the hood.
• Support of Redis Cluster mode and multi-key commands in cluster mode.

Obstacles along the way

One of the challenges that we faced while picking a library is that, for the aforementioned third point, most libraries didn’t seem to support multi-key commands in cluster mode, which are essential as they are widely used in our codebase (luckily, we don’t really care about atomicity when it comes to these kinds of operations). Thankfully, we found a suitable client library, and we were able to use it without introducing lots of changes to our code base. It’s important to mention that, in our case, it was relatively simple since our applications didn’t care about atomicity when it came to our multi-key operations, but in case the need arises in the future, we can always use hash tags, which can be used to force certain keys to be stored in the same hash slot. This should be used carefully, however, as it can easily skew the data distribution across hash slots and across nodes.

Moving Forward

As mentioned, we took this in steps. So we started by just deploying, taking out the old Redis library and replacing it with the one in the market service, which is much better supported, and making sure it worked well before switching to cluster mode.

Then, since we use AWS Elasticache for a managed Redis deployment, we created an ElastiCache cluster of 3 nodes with cluster mode enabled (using Terraform, which we use to define and deploy all our infrastructure) only for the market service. But when we started applying this in code on the market service we faced an issue along the way…

More Obstacles

Turns out Redis Cluster is not supported in the Datadog APM library that we use, and since monitoring and observability are two key components to anything that we do at Thndr, we couldn’t move forward. We faced a dead end and were contemplating what to do. Should we wait until it’s supported? Should we fork the library and build support for it ourselves? The truth is, we were really tight on time, and we were expecting a substantial jump in our app load any day, so we had to find a solution that worked now.

Monkey patching to the rescue! Even though monkey patching is not really the most efficient or aesthetically pleasing code to write and maintain, it was a fast and effective option. We were able to monkey-patch the library and make sure monitoring was working on our staging environment in around 30 minutes.

Showtime and Results

And finally, we were ready to deploy Redis cluster mode in one of our services and see it in action on production. Immediately upon deploying our solution to production, the results were immediate. The following graph was taken during market hours, and it shows Redis latency mapped just to mere microseconds!

Wrapping up

To sum it up, we discussed the problem, discovery, solution, and implementation of one of Thndr’s app’s technical debt items that had become a higher priority due to the app’s phenomenal growth in daily transactions and user base. At Thndr, we are firm believers in continuous improvement and doing things the right way. We also acknowledge that there will always be trade-offs and that technical debt should be prioritized during each iteration and at every level. However, we also know that there are many amazing projects and codebases for products that were not fortunate enough to continue. Every day, trade-offs are inevitable!

How we reduced Redis latency for a better investing experience at Thndr! was originally published in Thndr on Medium, where people are continuing the conversation by highlighting and responding to this story.