Why Tests Matter in Software Engineering
Integration tests are essential for verifying real-world interactions between system components. However, running and maintaining them — especially with Docker-based setups — often leads to slow feedback, flakiness, and complex configuration headaches.

Wix engineers tackled this challenge and developed a new approach that cuts testing time by more than 50x, all while streamlining the workflow for developers.

Highlights from the Wix Solution

• In-Memory Integration Testing:
  Wix replaced external dependencies (like databases and RPC services) with in-memory implementations for testing. This avoids slow Docker containers, reduces infrastructure overhead, and increases reliability.
• Hexagonal Architecture:
  Business logic is decoupled from infrastructure concerns using well-defined ports and adapters. This makes it simple to switch between real and in-memory collaborators in tests, preserving the consistency between test and production code.
• App Context Builder:
  Dependency wiring is managed automatically, ensuring test environments stay in sync with production without extra developer effort.

Benchmarks and Impact

• Nearly 100% of new Wix integration tests now use this method.
• Feedback cycles are much shorter, making it easier to detect and fix issues early.
• Flakiness and configuration errors common in Docker-based tests are largely eliminated.

Read the Full Engineering Deep-Dive

Curious about the architectural patterns, implementation details, and performance benchmarks behind this approach?

Check out the full engineering post here:
50x Faster and 100x Happier: How Wix Reinvented Integration Testing

Scaling a platform as expansive as Wix is no small feat. As a leading website-building platform catering to millions of users globally, the need to handle enormous traffic and vast datasets is ever-growing. Today, we delve into the battle-tested strategies Wix employs to tackle these scaling challenges and ensure a smooth, reliable service for our users.

By the end of this article, you’ll have a clearer understanding of which strategies might work best for your own use-case.

A Glimpse into Wix’s Scale

To understand the magnitude of our operations:

• Our system comprises 4,000 microservices
• We handle 4 billion HTTP transactions daily
• We produce 70 billion Kafka messages every day
• Our systems manage over 4 petabytes of data

Operating at such a colossal scale inherently comes with unique challenges. Our infrastructure must not only accommodate today’s demands but also scale to meet future growth as more users join our platform.

Vertical vs. Horizontal Scaling

When it comes to microservices scaling, there are two basic options:

• Vertical Scaling (Scaling Up): Increasing the power of an existing machine by adding more CPU, RAM, etc.
• Horizontal Scaling (Scaling Out): Expanding the system by adding more machines to handle the load.

Initially, vertical scaling might seem straightforward. However, real-world constraints quickly emerge, such as bottlenecks and escalating costs.

For instance, our Redis machine used by our rate limiting service frequently hit 100% CPU utilization. This caused disruptions and triggered fallbacks to simpler algorithms. Despite upgrading to more powerful CPUs multiple times, we still faced these issues.

This clearly illustrates the limitations of vertical scaling. At a certain point of higher traffic loads, all Redis users need to move to a cluster of Redis machines, i.e., horizontally scale Redis usage.

The Horizontal Scaling Paradigm

To effectively manage our scaling needs, we’ve shifted towards horizontal scaling. This approach involves distributing workloads across multiple machines, significantly enhancing system responsiveness and reliability.

It’s more efficient for distributing traffic and preventing bottlenecks, but it requires complex management to ensure coherent operation across all machines. Horizontal scaling requires careful selection of routing/sharding keys (how data is managed and routed).

Routing and Sharding Strategies

Efficient data management across different nodes requires effective routing and sharding strategies. These strategies aim for uniform data distribution and consistency, thereby reducing latency and boosting overall performance by distributing data and queries effectively.

Fixed Routing

In fixed routing, data or traffic is assigned to specific shards or paths based on predefined keys. This method helps maintain consistency and predictability.

Dynamic Routing

Dynamic routing adjusts the distribution of data or traffic on-the-fly based on custom rules or mathematical functions. This method offers flexibility and scalability, particularly for applications with fluctuating demand. For instance, we employ hashing functions to achieve uniform distribution in a consistent manner.

Highlighting Wix Scaling Examples

As Wix scales, we face various challenges in managing high loads, ensuring reliability, and maintaining performance consistency across different services. Here’s how we address these challenges with our routing and sharding key selection strategies:

1. Kafka Infra Services Fixed-Key Scaling Strategies

Wix operates multiple proxy services that handle Kafka traffic. Here, we will focus on two critical services: the Wix Kafka consumer proxy, which caches messages and reduces broker connections for improved efficiency and performance, and Wix’s Kafka-to-HTTP-webhooks translation service, which exposes events to external apps running on the Wix platform.

Managing high volumes of Kafka traffic through these services presents significant challenges:

Pain Points:

• Single Points of Failure: An issue with even one Kafka connection can potentially bring down the entire service, as the impact can be significant in a single service deployment. This includes issues like CPU or memory exhaustion, which can propagate and affect overall service performance.
• Noisy Neighbors: Maintaining high reliability and performance for each individual tenant is challenging when all service instances handle requests from all tenants, making it difficult to isolate issues impacting one tenant from affecting others.

Our Solutions:

To tackle these issues, we implement horizontal scaling with straightforward fixed-keys traffic routing:

• Fixed Clusters Keys Routing: We deploy consumer proxies for each Kafka cluster. This strategy avoids single points of failure by ensuring efficient message consumption and distributing the load evenly among consumers.

• Fixed QoS Groups Keys Routing on Kafka Dimension: Our Kafka-to-HTTP-webhooks translation system uses QoS tiers to segment data. By doing so, we ensure that higher QoS tenants maintain reliability even when other segments experience frequent timeouts.

Currently, this fixed-keys traffic routing method effectively ensures high availability and performance for all tenants. However, we are considering introducing more dynamic key selections based on live metrics such as traffic and latency to further enhance our capabilities if needed in the future.

2. Web Traffic Management

Fixed Segments: At Wix, web traffic is divided into two fixed segments:
the Editor segment (site editing and business management) and the Public segment (serving sites to visitors). This decoupling — driven by the need to transform data structures during the publishing process — allows us to tailor service levels for different functions.

The Public segment, affecting all users, demands the highest service levels regarding availability, performance, risk of change, and recovery time. The Editor segment, impacting fewer users, can operate with lower service levels, allowing for greater agility.

Dynamic Routing: Load balancers dynamically route traffic across multiple servers. For instance, nginx routes *.wix.com traffic to the Editor segment and wixsite.com or user domains to the Public SSR segment. This ensures efficient web request management and seamless performance during traffic spikes.

3. Reactions App Using Kafka & DynamoDB to Scale

Reactions are a key feature for social interaction on Wix sites and mobile apps, including likes, stars, thumbs up, comments, and more. To support millions of users, the system needs to be fast, reliable, and scalable.

Here’s how we address the key design considerations:

• High Write Throughput: Many users will post reactions simultaneously, requiring a system that can handle a high volume of concurrent writes.
• Low Latency Reads: Users expect to see reactions in real-time, necessitating a system that provides immediate feedback.
• Scalability: As Wix grows, the reactions system must scale seamlessly to accommodate increasing loads.

Dynamic Routing on Kafka Dimension
For the Reactions app, we use Kafka’s partitioning and message-handling capabilities to manage high loads efficiently.

• Partitioning: Divides topics for parallel processing and horizontal scaling.
• Performance & Durability: Ensures high throughput, low latency, and reliable data storage.
• Order Guarantee: Maintains event order within each partition.
• Scalability: Sharding patterns and increasing partitions enable seamless growth.
• Dynamic Message Routing: Messages are dynamically produced to different partitions based on their keys. This allows for balanced load distribution and ensures efficient processing.
• Consumer Parallelism: Consumers in a group are assigned different partitions. To scale, increase the number of partitions.

By dynamically routing messages to partitions, we ensure the Reactions app processes real-time interactions efficiently, supporting high write throughput and low latency reads.

Dynamic data Sharding (DynamoDB):
DynamoDB uses tables with primary keys for data distribution. Composite primary keys (partition and sort keys) enable optimized data access and the storage of multiple items with the same partition key.

• Partition Key Importance: Distributes data across storage partitions, balancing load and scaling the database.
• Schema Flexibility: Allows each item to have unique attributes.
• Capacity Limits: Each partition holds up to 10 GB of data and supports up to 3,000 read or 1,000 write capacity units.
• Internal Hash Function: Uses the partition key as input to a hash function to determine storage partitions, ensuring efficient data retrieval and load distribution.

These advanced sharding techniques allow DynamoDB to dynamically adjust to data and throughput demands, maintaining consistent performance. This scalability ensures the reactions system can seamlessly grow alongside Wix.

4. Data Locality for Enterprise Customers

Fixed Data Sharding
For enterprise customers with specific data locality requirements, Wix employs fixed geographic sharding to comply with regulations like GDPR. This ensures that data, such as employee-related information, is stored only in specified regions (e.g., only in the EU or the US). Wix has data centers in both the US and Europe and supports tenants in choosing where their data is persisted. Some tenants may opt for regional storage while others prefer global copies. Most of Wix’s microservices data is stored in MySQL tables.

Dynamic Data Routing
Using ProxySQL, a high-performance load balancer for MySQL, Wix dynamically routes database traffic while maintaining geographic sharding. ProxySQL allows for:

• Intelligent Query Routing: Distributes database traffic for optimal load balancing and performance.
• Enhanced High Availability: Redirects traffic away from failed nodes to maintain uninterrupted database access.
• Sharding Support: Manages large datasets and scales out by routing queries to the appropriate shards.

ProxySQL enables configuring custom query rules to route specific SQL traffic to predefined groups of nodes within a MySQL cluster. Routing decisions are made based on regular expressions in SQL statements, ensuring efficient performance and compliance with data locality requirements.

Key Takeaways

Implementing horizontal scaling comes with increased complexity in system modeling, architecture, and management. However, these strategies are essential to ensuring our platform remains fast, reliable, and scalable, meeting the diverse needs of our growing user base.

• Choosing Keys to Split: Carefully analyze the workload and traffic patterns to select the most effective keys for sharding and routing.

• Tech Stack Choice: Select a tech stack that supports efficient scaling strategies. Technologies such as Kafka, DynamoDB, and ProxySQL offer robust capabilities for horizontal scaling.

Overall Scaling Strategy Selection

The following diagram illustrates a decision tree to guide you in selecting the optimal scaling strategy for your use case. This decision tree focuses on three key considerations:

1. Resource Usage and Cost: Evaluate the resources required and the associated costs of different scaling strategies.
2. Number of Potential Tenants: Consider the number of different tenants your system needs to support.
3. Complexity of Potential Routing Rules: Assess the complexity of the routing rules needed to manage traffic and data effectively.

Conclusion

Scaling a platform as vast as Wix requires an ongoing commitment to innovative solutions and flexible strategies. By blending vertical and horizontal scaling with fixed and dynamic routing and sharding techniques, Wix optimizes performance, adapts to evolving requirements, and ensures a reliable service.
This approach equips Wix to handle current demands while preparing for future growth, and it can provide valuable insights for anyone facing similar scaling challenges.

Thank you for reading!

If you’d like to get updates on my future software engineering blog posts, follow me on Twitter, LinkedIn, and Medium.

You can also visit my website, where you will find my previous blog posts, talks I gave in conferences and open-source projects I’m involved with.

If anything is unclear or you want to point out something, please comment down below.

Further reading:

• Leader election and Sharding Practices at Wix microservices
• How to choose the right database for your service
• 4 Microservices Caching Patterns at Wix
• 6 Event-Driven Architecture Patterns — Part 1

Scaling Horizons: Effective Strategies for Wix’s Scaling Challenges was originally published in Wix Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Managing Kafka event schemas for 2000 microservices can be quite challenging and painful. Potential challenges include:

1. How can you maintain safe schema evolution and avoid catastrophic poison pills that will bring down your Kafka consumers?
2. How can you make schemas easily discoverable and avoid wasting precious dev time?
3. How can you enforce good documentation to avoid usage mistakes and severe data corruption?
4. How do you automatically sync changes in schemas across different tech stacks to avoid duplication mistakes and stale versions?

Over the past few years, Wix has experienced tremendous growth in both scale and complexity.

From a WYSIWYG visual website editor to a complete ecosystem of features that empowers self-creators and design agencies to create professional websites, and allows anyone from individuals to large enterprises to manage their online presence and business.

Intro article: How Wix Accelerated Open Platform Dev with Standardized APIs & Schemas

However, this growth introduced a technical challenge in managing a growing list of event schemas (for Kafka) and API designs (for gRPC) from hundreds of services.

Wix developers and API experts have implemented various tools and practices to make sure all the above challenges are met, dev velocity is kept high, and production issues are kept to a minimum.

Wix schemas are defined with protobuf

Previously, the standard schema definitions at Wix backend were written as Scala case classes serialized to JSON. It was mainly used for RPC as well as Kafka payloads (At the time, Kafka was less heavily used for microservices). Custom JSON-based de/serializers were created for the JVM and Node.js services (as javascript became a second language for wix backend).

Enter gRPC and protobuf
As Wix scale grew tremendously, both in terms of traffic (500 Billion HTTP requests per day), but also in terms of product areas (many verticals such as hotels, restaurants, bookings, etc…) and developers (>1000 devs), a standard microservice API architecture was introduced based on gRPC communication and protocol buffers (aka protobuf) IDLs.

Why Protobuf
Tools that generate client and server code from protobuf definitions are available for many languages, offer fast parsing, and allow forward compatibility in many cases.

In order to avoid different standards for gRPC and Kafka, Protobuf was also used to describe Kafka event schemas and not only gRPC services.

Discovering Schemas

Schema Registries like Confluent’s or Red Hat’s allow you to store your schema in one central location. This can provide many benefits such as:

• Central location to track all schemas used in production
• Document the data format required for each topic
• Centralized control of schema evolution

In addition, they offer an API for Kafka producers & consumers and other clients (such as cloudEvents) to fetch the current version of the schema in order to understand how to serialize or deserialize the events.

Wix Docs
At Wix, we have our own web product for discovering and presenting schemas called Wix Docs. It is used for Wix third-party application development.

Screenshot of Wix publicly available API (REST, webhooks, SDKs)

Wix Docs utilizes build tools (e.g. Bazel) plugins, which collect API & Schema information from .proto, documentation.yaml and .md files.

The plugin generates OpenApi swagger.json files and sends them to Wix Docs services to be able to store and view the documentation.

Automatic Schema discovery & updates
In addition to Wix Docs, Wix also has a service called Business Schema that provides a list of business-entities and a machine-readable “API descriptor” for each Domain Entity with its actions and events. Business-schema API and events allow Wix developers to build client libraries, IDE plugins, and other tools that dynamically interact with Wix APIs.

Serializing Schemas at Wix

Wix Kafka Client “Greyhound” takes care of serialization out-of-the-box
Kafka producers and Kafka consumers are not used directly in Wix microservices.

Instead, an additional layer called Greyhound is used. Greyhound seeks to provide a higher-level interface to Kafka and to express richer semantics such as parallel message handling or retry policies with ease.

Open Source Greyhound Schema flexibility
The open-source Greyhound producer and Greyhound Consumer RecordHandler are flexible and allow for any type of payload de/serializer

With Open Source Greyhound, a Schema Registry can be used, the same way it would have been when directly using Kafka producers and consumers, in order to fetch the current schema. The Kafka provided SerDe can be wrapped inside the Greyhound SerDe and configured for Producer and RecordHandler seen above.

Wix Greyhound default JSON Serializer/Deserializer
For Wix needs, Greyhound has an additional dedicated layer (which is not open-sourced).
The “Wix” producer and consumer API provide out-of-the-box support for SerDe (Serializer/Deserializer).

While it is possible to override the SerDe, defaults are used for 99.9% of use cases.

For the Wix producer, the default serializer accepts protobuf-derived Scala case classes and serializes them with JSON protocol to bytes.

For the Wix consumer, the default deserializer does exactly the reverse, accepts bytes and deserializes them with JSON protocol to protobuf-derived Scala case classes.

Schema Evolution & Validation

Confluent Schema Registry provides compatibility checks as schemas evolve. Each Schema change is assigned a unique version with validation done according to the configured compatibility type for the schema. The main types are BACKWARD (the default) and FORWARD.

When validation is turned on for a topic, the registry will not allow producers to produce a message according to a schema that will fail to be consumed because of backward or forward compatibility issues.

Wix Validation
Most of the schema validations at Wix are the responsibility of the developer, to make sure that they don’t introduce breaking changes. There are extensive internal API lifecycle guidelines that describe how Schemas should be introduced and changed (Including backwards compatibility, maturity levels, and versioning.)

Automatic validation
In order to help developers and avoid compatibility issues, a dedicated tool was developed in order to make sure there are no breaking changes in proto files.

Building and Syncing Schemas inside CI/CD pipeline

There are 2 major build systems at Wix.
One is based on Bazel for JVM languages and services, and the other one is called Falcon (internal build tool) for Java/TypeScript and Node services.

Each has its own ways of sharing and compiling protobuf schema definitions.

Intra Bazel/Scala dependencies
Bazel dependencies can potentially be compiled from scratch (no semantic versioning!). For proto file dependencies between two Scala services built by Bazel, all compiled artifacts (including Scala jars created from proto files) are stored in caches to improve build performance. Recompilation is avoided as long as the proto files haven’t changed.

Bazel build targets
Protobuf build targets that collect all of a service’s proto files together are defined as follows:

wix_proto_library targets are actually macros that encapsulate creation of the original proto_library and also proto_npm_package for creation of NPM packages for interoperability with Node.js services.

Jars including the generated Scala files are the output of wix_scala_proto_library

Intra Node dependencies
Node services rely on an in-house library called wix-proto-codegen that scans proto NPM package dependencies (in package.json) and generates JavaScript and TypeScript classes for it.
It utilizes the protobufjs npm package as its encoding engine.

package.json declaration
Usually wix-proto-codegen is run via package.json as follows:

A gRPC client is automatically generated after dev adds proto npm package as a dependency.

Scala->Node.js dependencies
In order for proto files originating in the Scala CI system to be consumable by the Node.js CI system a dedicated service called proto-sync service was created.

proto-sync listens to build events coming from Bazel via another Wix service, filters only proto targets and triggers a build event in the Node.js CI server. (The build artifact will be created first, if it hasn’t yet.)

This build event will create a standard NPM module (with upgraded version) and will upload it to the NPM private registry.

Soon this flow will change and NPM packages will be generated directly from Bazel build.

Node.js->Scala dependencies
In order for proto files originating in the NPM registry to be consumable by the Bazel build tool, they have to be copied to a dedicated GitHub repository and Bazel build files have to be created alongside them. Now they can be consumed as a source-dependency by other Bazel targets.

Summary

While popular Schema Registry solutions for Kafka like Confluent’s can be very helpful with Schema management, it didn’t fit Wix’s unique needs and circumstances (Supporting gRPC, Kafka, and derived external APIs in one holistic standardized way).

Wix infrastructure and Devex teams have created an extensive array of frameworks and tools to support Schema management and development velocity. These Include automatic discovery, serialization, and validation.

Meeting the challenges
Challenges included multiple technological stacks and many legacy services that required support and complex migration.

All challenges were handled successfully as an organizational mindset change was made to treat every single schema very carefully and thoughtfully as it will most probably be consumed by developers external to Wix.

Thank you for reading!

If you’d like to get updates on my future software engineering blog posts, follow me on Twitter, LinkedIn, and Medium.

You can also visit my website, where you will find my previous blog posts, talks I gave at conferences and open-source projects I’m involved with.

If anything is unclear or you want to point out something, please comment down below.

Further reading:

• How Wix Accelerated Open Platform Dev with Standardized APIs & Schemas
• Troubleshooting Kafka for 2000 Microservices at Wix
• Event Driven Architecture — 5 Pitfalls to Avoid

How Wix manages Schemas for Kafka (and gRPC) used by 2000 microservices was originally published in Wix Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Over the past few years, Wix has experienced tremendous growth in both scale and complexity.

From a WYSIWYG visual website editor to a complete ecosystem of features that empowers self-creators and design agencies to create professional websites, and allows anyone from individuals to large enterprises to manage their online presence and business.

However, this growth introduced a technical challenge in managing a growing list of disparate API designs and event schemas from hundreds of services.

Wix developers and API experts have implemented various tools and standardized practices to make sure the above challenge is met, dev velocity is kept high, and production issues are kept to a minimum.

Standard APIs & Event Schemas — Standardization effort

In the early days of Wix services, most of them were monoliths, with APIs based on internal use cases only and not exposed to 3rd party developers.

In order to incorporate standardization and cultivate the “open platform” mindset, It was important to move from feature-based APIs (that mostly frontend uses) to domain-based API designs and event schemas.

Designing part of a platform
The idea was to define a service’s main entities, then define the APIs and events and also define how everybody would use them, looking not only at your immediate use cases, but also research the API as a product. Talk to potential 3rd party apps developers, go over similar APIs that exist in other companies.

This allowed for better efficiency throughout the company, as APIs and events were re-used instead of duplicated. At the same time, common infrastructure and terminology were formed.

For more on adopting an API first approach you can read this article.

Exposed externally
Wix now has a comprehensive developer platform that enables the extension of Wix websites and back-office capabilities. The platform consists of two parts:

1. Extending websites with Velo — a JavaScript based development environment for both frontend and backend code.
2. Wix App Market for third-party applications.

Most of Wix’s microservices internal gRPC APIs are exposed as REST APIs via automatically generated thin wrappers for applications and also Velo APIs.

Wix’s microservices Kafka Events are now automatically open to external developers via Webhooks for applications and Velo Events .

Wix schemas are defined with protobuf

Previously, the standard schema definitions at Wix backend were written as Scala case classes serialized to JSON. It was mainly used for RPC as well as Kafka payloads (At the time, Kafka was less heavily used for microservices). Custom JSON-based de/serializers were created for the JVM and Node.js services (as javascript became a second language for wix backend).

Enter gRPC and protobuf
As Wix scale grew tremendously, both in terms of traffic (500 Billion HTTP requests per day), but also in terms of product areas (many verticals such as hotels, restaurants, bookings, etc…) and developers (>1000 devs), a standard microservice API architecture was introduced based on gRPC communication and protocol buffers (aka protobuf) IDLs.

Why Protobuf
Tools that generate client and server code from protobuf definitions are available for many languages, offer fast parsing, and allow forward compatibility in many cases.

Both RPC and REST
Google provides protobuf http extension that enables the dev to define http mapping per RPC method. It also provides JSON-serializers to/from scala case-classes that were auto-generated by protobuf. This uncovered the ability to use a single schema definition to model both RPC and REST APIs without any additional translation infra.

Event-driven architecture & Domain Events

In the last couple of years, emphasis started to shift to a more event-driven microservices architecture approach with the introduction of domain events.

A common standard was defined for sending events from services related to a specific domain (e.g. product-changed) in order to ensure a streamlined approach for clients to consume events.

This setup serves both Wix consumers (using Protobuf serialized with JSON and streamed via Kafka), and external consumers (via webhooks and Velo events).

Domain events
Domain events are meaningful business events notifying of a change in the state of the domain entity. Examples:

• Entity created
• Entity updated
• Entity deleted
• Email sent to member
• Payment transaction started

The domain events common proto defines an envelope with common metadata in order to make it easier for infrastructure and “horizontal” services to consume them (more on this in the following section).

Discovering APIs & Schemas with Wix Docs

At Wix, we have our own web product for discovering and presenting schemas called Wix Docs. It is used for Wix third-party application development.

Wix Docs utilizes build tools (e.g. Bazel) plugins, which collect API & Schema information from .proto, documentation.yaml and .md files.

The plugin generates OpenApi swagger.json files and sends them to Wix Docs services to be able to store and view the documentation.

Automatic API & Schema discovery & updates
In addition to Wix Docs, Wix also has a service called Business Schema that provides a list of business-entities and a machine-readable “API descriptor” for each Domain Entity with its actions and events. Business-schema API and events allow Wix developers to build client libraries, IDE plugins, and other tools that dynamically interact with Wix APIs.

Schema Evolution & Validation at Wix

Most of the schema validations at Wix are the responsibility of the developer, to make sure that they don’t introduce breaking changes. There are extensive internal API lifecycle guidelines that describe how Schemas should be introduced and changed (Including backwards compatibility, maturity levels, and versioning.)

Because of the API-first “Open Platform” mentality, all developers have built the mindset to make sure they change their public APIs and schemas in a safe manner.

Automatic validation
In order to help developers and avoid compatibility issues, a dedicated tool was developed in order to make sure there are no breaking changes in proto files.

Version Migration
A typical schema version lifecycle at Wix goes through these stages:

1. Create the model
2. Stabilization
3. Add features / abilities / enhances / fixes
4. Deprecation
5. End-of-life

In case a breaking change must be introduced, the old version is marked as deprecated, and a new version is created.

Summary

In order to reach the “Open Platform” vision, Wix has created detailed guidelines and an extensive array of frameworks and tools to support Schema management and development velocity. These Include automatic discovery, serialization, and validation.

Mindset Change
An organizational mindset change was made to build a truly open platform that treats every single schema very carefully and thoughtfully as it will most probably be consumed by developers external to Wix.

Higher dev velocity
As a testament to this success, the rate of new business microservices creation and overall event consumption has risen dramatically in the last few years.

Despite having over 2000 event-driven microservices in production, Wix is able to onboard a new business domain and integrate it with the surrounding ecosystem, consuming all the necessary APIs and events, in just a few days.

A true open platform
In addition, this new domain is then automatically open to third-party application developers and Wix website developers, creating one seamless experience for both internal & external developers throughout the lifetime of this API & Schema version.

Thank you for reading!

If you’d like to get updates on my future software engineering blog posts, follow me on Twitter, LinkedIn, and Medium.

You can also visit my website, where you will find my previous blog posts, talks I gave at conferences and open-source projects I’m involved with.

If anything is unclear or you want to point out something, please comment down below.

Further reading:

• How Wix manages Schemas for Kafka (and gRPC) used by 2000 microservices
• 5 tips for writing great client SDK libraries
• 5 Database technologies used by 2000 Wix microservices
• Event Driven Architecture — 5 Pitfalls to Avoid

How Wix Accelerated Open Platform Dev with Standardized APIs & Schemas was originally published in Wix Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

How do you understand why your service’s database has some incorrect values? (And increase your org dev velocity?) How do you get to the root cause and reduce debugging time? (And save money and effort?) Event processing sometimes fails or is extremely slow and a consumer lag builds up, how do you bring it back down?

In the past, Wix developers had experienced stressful, anxiety-filled moments where they needed to fix some urgent production issue related to the event streaming layer, but they were blind to the root causes and powerless to fix them in a straightforward fashion. Instead they were required to drop processing of many events or execute time-consuming DB fixing scripts.

In the last few years my Data streams team, in charge of the event-driven infrastructure on top of Kafka, created many tools and services in order to give the full power and visibility to Wix developers to easily troubleshoot and fix event-streaming related production issues. We have seen the incident resolution time dramatically decrease with more pinpoint solutions that avoid costly database repairs.

Following are 5 must-have-features to easily troubleshoot and fix event-streaming related production issues:

1. Trace events flow

Ecommerce flow
Consider the classic ecommerce flow, where a user places an order and triggers a chain of events:

• OrderCreated event is produced by order service
• Payment service consumes the order events, performs payment and produces a paymentCompleted event
• Inventory service consumes paymentCompleted event and produces inventoryUpdated event

Of course, the actual Wix ecommerce system is much more complex and has many different events flowing between many different microservices.

Data corruption
At some point Wix developers noticed that the inventory service database does not reflect the actual inventory levels of some products.

Even with this simplified example, you can start seeing how hard it is to track the events flow and get a clue for the root cause of the issue. Was the OrderCreated or PaymentCompleted event produced correctly? Do they contain all relevant products? Or maybe the inventory service failed to process the PaymentCompleted event correctly?

Without proper tracing tools, developers were left searching logs of each service for clues, or manually streaming many events in multiple topics in the hopes of finding the potentially faulty events.

Request Ids Propagation
In order to easily trace the flow for each request throughout ecommerce services, we put the original requestId in the event headers. These headers are then deserialized and written as extra metadata to monitoring and logging infrastructure. The requestId is also put in headers for new produced events which recursively allows to propagate this metadata in all participating services.

This “Context propagation” is provided out-of-the-box for all Wix service by Greyhound, a JVM library and polyglot sidecar that forms a data streaming abstraction layer on top of Kafka with many features that help developers with features that Kafka itself lacks, like produce/consume retry strategies and higher performance parallel and batch consumers.

By having the request Id attached to all logs and metrics, we can easily see in Grafana dashboards which services successfully processed events related to this requestId:

Logs are also easily filterable by the requestId, so you can see relevant logs for this event in all services at once, including the logs of the relevant inventory database changes.

Open Telemetry
In addition to placing requestId in Kafka record headers, Greyhound also exports OpenTelemetry traces and spans information. While the data is currently available for each service internally, In the near future, full distributed tracing information will be provided, including full breakdown of operations

2. Easily lookup a specific event payload

Continuing Ecommerce data inconsistency investigation
Let’s continue the ecommerce example from the tracing segment above. We want to investigate why the Inventory service database does not reflect the actual inventory levels of some products.

Let’s say we are satisfied that all events were successfully produced and consumed by looking at the tracing dashboards for the suspicious user requests. Next we want to investigate if all products were accounted for in the OrderCreated and PaymentCompleted events related to the suspicious user requests.

Kafka Control Plane Apps
In order to do that we need to look at the relevant event payloads. Some Kafka control plane applications allow you to stream and filter events, including UI for Apache Kafka, Confluent Control Center, and Conduktor. You can also use Kafka CLI tools to find relevant events.

Wix Custom Control Plane — Event Stream View
At Wix, we created our own custom control plane, in order to utilize the power of Wix’s Greyhound Platform’s metrics and configuration. One of the first features we implemented was the ability to stream events from specific topics. The user first chooses which region and Kafka cluster he wishes to stream from. They can also choose to filter the stream, to start from some time in the past, and/or from some partition or offset. They can also specify specific text to search for in Kafka record key, payload or headers.

In our ecommerce Inventory investigation case, we can search for the suspicious requestIds and look inside their payloads and check if all the items are accounted for.

Duplicate processing and Idempotency
As it Turns out, the events had all the items in place, but upon closer examination of the logs, it was discovered that the Inventory service processed the same events twice. Because of some partial failure during processing that happens from time to time. The service developers didn’t make sure to include deduplication protection in their database updates and make them idempotent.
For more information on how to introduce idempotency to your service, you can review the 5th pitfall from one of my previous articles.

3. Investigate “slow” consumers root cause

Consumer Lag Alert
At Wix, once a consumer fails to keep up with the amount of produced events for more than 30 minutes, an alert is triggered in Grafana and the consumer owning team is notified in Slack. For Kafka this phenomenon is called Consumer Lag (for queue based brokers such as RabbitMQ it’s simply called a large and growing queue.)

Inventory Service Lag
Let’s go back to our ecommerce example. A consumer lag is building up for the Inventory service Payments topic consumer. The developers are starting to be worried that the inventory database will not reflect the reduced inventory due to recent completed purchases.

Produced/Consumed Rate Grafana graph
As they start to investigate the consumer lag following the Slack alert, WixStores developers go to the Inventory service Grafana dashboard and see the difference in the throughput between producer and consumer for the Payments topic. While the produce rate is ~4K RPM, the consume rate is only ~1K RPM.

“Stuck” Consumers Grafana graph
Another very important graph that Inventory Service devs can view is called Longest Running Consumer Handler, where they can see if any/all consumer instances are “stuck” on handling.

This graph is made possible by Greyhound, our Kafka wrapper library and service. Greyhound exposes a Prometheus gauge for each topic consumer handlers, such that it measures the currently longest running consumer handler (the longest running provided lambda function) running in this pod. For more on how Greyhound works, you can check out this video.

Restart “stuck” Kubernetes pod
Back to the investigation. According to the above Grafana graph, it seems like the Inventory Service Payments topic consumer is stuck in one of the Kubernetes pods. The developer decides to restart the pod in the hopes that the issue will go away and Payments topic consumption rate will go back to normal.

But alas, it doesn’t help. The consumer rate stays lower than the producer rate. But not to worry, there are other monitoring tools she still hasn’t used.

Consumers Lag View in the Wix Kafka Control Plane
In order to help the developers investigate the root cause of consumer lags, we created a dedicated view in our custom control plane where the developers can easily see which Kafka topic partitions have the most lag.

In the ecommerce inventory case, after the pod restart, the lag kept growing for the same partition, just with another pod being assigned to it.

View “last” event
In case the lag is mostly found in 1 partition, the devs can click on “Last Message” button and view the current event being processed, to understand if its payload or headers may cause the issue

After reviewing the event payload, the developer understood that the event includes a huge amount of items, which the Inventory service fetched to memory and caused the Kubernetes pod’s service container to run out of memory, which caused the JVM process to become unstable and the consumer handler to get “stuck” and lag to build up.

4. Skip or replay events

Continuing with our ecommerce example, in the last segment, we reached a situation where the Payments topic Inventory service consumer lag was caused by a “rogue” event with a huge payload.
The developer decided that the best course of action would be to just skip the event, as she saw in the stream view (described above in section 2) that the issue is isolated to just this one event. It will also mean she can fix the underlying issue in a more relaxed setting, and then replay the event once the bug is fixed.

How to skip or replay events in Kafka?
1. Kafka SDK seek API
While Kafka allows consumers to skip or replay events by using the seek api, it’s not realistic to expect that the consumer code will know in advance the exact scenarios when it is needed to skip events, how many events, and so on.

2. Kafka-consumer-groups.sh command
In the past, we utilized kafka-consumer-groups.sh cli script in order to change the consumer group current event offsets, but it required shutting down all consumer pods, so they will be able to start again with the newly configured offsets.

Another disadvantage is that it’s not easy and straightforward to work with this command line interface, so it was mostly up to our streaming infra dev team to execute these skips/replays on behalf of other backend developers at Wix.

3. Change consumer group name
In cases where it’s possible to drop all unprocessed events, another option is to change the consumer group name, and start processing events “from latest” offsets.
If replaying all available events is possible, the configuration of “from beginning” can be set

4. Easily request Greyhound to skip or replay events
Recently, my streaming infra team developed a way to request from Greyhound, our Kafka wrapper library and service, to skip or replay events.

Currently the request is done via RPC, but soon there will be a new Skip/Replay events view in our Wix Custom Kafka Control Plane.

The user requests to change a consumer group’s offsets for one or more partitions.
Either with the explicit offset value, or from some timestamp.

1. The inventory service configuration is changed to include the new reset-offsets configuration.
2. The configuration change causes the service pods to restart and the consumers to start from the requested offsets to start from.
3. An additional mechanism makes sure that the reset will not happen again on subsequent restarts, but its details are not in the scope of this article.

Inventory service consumer group lag finally drops
Once the Inventory service consumer group successfully skips the problematic event after the developer requests it from Greyhound, the consumer group lag starts to drop and the production issue is resolved successfully.

5. Redistribute Single Partition Consumer Lag

There are times where a consumer lag is caused by a misconfigured Kafka event key. When an event is produced with a key, all events that share the same key are produced to the same partition and thus are processed sequentially.

Events with the same key produced to the same partition. Each Partition is processed sequentially by a different Consumer pod (C1..C4)

Misconfigured Kafka event key
Continuing with our ecommerce example, in Black Friday 2021, the Payments topic Inventory service consumer lag was caused by relatively slow consumer handlers, but also because of problematic choice of Kafka event key by the payments service. The key was configured to be the vertical app id which means small distribution such that many events go to the same partition instead of utilizing all the partitions of the topic.

Quicker lag resolution
Involving the developers of payments service to change the key and deploy a new version took a long time and caused the issue to remain unresolved for a few hours. So as a lesson, Wix streaming infra dev team decided to develop a feature to redistribute the events that cause the single partition lag among all partitions.

Out-of-order processing
While redistribution of the events can cause out-of-order processing, for critical situations where the lag must be removed quickly, or in cases where order doesn’t matter to this particular consumer, this type of remediation can be quite useful.

In December 2022 we’re still prototyping the solution in production. I will update this article with our full robust battle tested solution once it becomes available.

Summary

Investing in good infrastructure for troubleshooting event streaming Apache Kafka related production issues is critical for being able to scale an event-driven architecture for your distributed environment of microservices.

Make sure to have the following monitoring capabilities:

1. Trace events flow
   Ascertain which services successfully processed events related to a user request and which haven’t by using OpenTelemetry protocol for example
2. Easily lookup a specific event payload
   Make sure event payload is not corrupted or the root cause of some unforeseen issue
3. Investigate “Slow” Consumers root cause
   Have relevant metrics such as longest running handler, and lag-per-partition view in place

Monitoring features have to be accompanied by remediation tools such as:

1. Easy events skip/replay
   Have the ability to skip events to remove consumer lag, or replay events to fix database issues
2. Redistribution of single partition lags
   Have the ability to redistribute single partition lags in order to resolve slow consumption due to badly distributed kafka event key

These capabilities: Trace, Lookup, Longest-Running, Skip, Redistribute, form the Acronym TLLSR

Having infrastructure for microservices event-streaming communication (like Wix has Greyhound and a custom Control Plane) can get you closer to your goals of higher dev velocity and reduced costs for maintaining a scalable and resilient event-streaming architecture.

There are several open source (UI for Apache Kafka) and commercially available tools (such as Confluent Control Center and Conduktor) that can help you get there faster.

Thank you for reading!

If you’d like to get updates on my future software engineering blog posts, follow me on Twitter, LinkedIn, and Medium.

You can also visit my website, where you will find my previous blog posts, talks I gave in conferences and open-source projects I’m involved with.

If anything is unclear or you want to point out something, please comment down below.

Further reading:

• How Wix manages Schemas for Kafka (and gRPC) used by 2000 microservices
• Event Driven Architecture — 5 Pitfalls to Avoid
• 6 Event-Driven Architecture Patterns
• 4 Microservices Caching Patterns at Wix
• Leader election and sharding practices at Wix Microservices
• Gwen Shapira on Event Driven Architecture and Cloud Native Kafka

Troubleshooting Kafka for 2000 Microservices at Wix was originally published in Wix Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Event driven architecture is very powerful and well suited for a distributed microservices environment. By introducing a broker intermediary, event driven architecture offers a decoupled architecture, easier scalability, and a much higher degree of resiliency.

But it’s much harder to set up correctly as opposed to request-reply client-server type architecture.

At Wix we have been gradually migrating our growing set of microservices (currently at 2300) from the request-reply pattern to event driven architecture over the last few years. Below there are 5 pitfalls that Wix engineers have encountered during our experimentation with event driven architecture.

These pitfalls have caused us great pain, in terms of production incidents, required re-writes and steep learning curves. For each pitfall I provide battle-tested proven solutions used at Wix today.

1. Write to db and then fire event without atomicity

Consider for example a simple ecom flow (we will use this example throughout this article)

Once the payment processing is done, the product inventory should be updated to reflect that the product is reserved for the customer.

Unfortunately, writing the payment-complete status to the database and then producing a “payment completed” event to Kafka (or some other message broker) is not an atomic operation. There could be situations where only one of the actions actually happens.

For example, cases such as the database being unavailable, or Kafka being unavailable, could lead to data inconsistency between different parts of your distributed system. In the case above Inventory levels may become inconsistent with actual orders.

Atomicity Remedy I — Greyhound resilient producer

There are several ways to mitigate this issue. At Wix we utilize two ways. The first one is our own messaging platform called Greyhound, that allows us to make sure the event is eventually written to Kafka via a resilient producer. One downside of this mitigation is that you can end up with out-of-order processing of events downstream.

Atomicity Remedy II — Debezium Kafka source connector

The second way to make sure both DB update action and Kafka produce action happen and that data remains consistent is using the Debezium Kafka connector. Debezium connector allows to automatically capture all the change events (CDC) that happen in the database (For MySQL via binlog) and produce them as Kafka events.
Kafka Connect together with Debezium DB connectors guarantees that events will eventually be produced to Kafka. In addition there is a guarantee that events order will be kept

Note that Debezium also works with other event streaming platforms such as Apache Pulsar.

2. Using Event sourcing everywhere

Event sourcing is a pattern where instead of updating an entity’s state upon a business operation, the service saves an event to its database. The service reconstructs an entity’s current state by replaying the events.
These events are also published on an event bus such that other services can also create materialized views on other databases that are optimized for queries by replaying the events.

While there are certain advantages to this pattern (a reliable audit log, performing “time travel” — the ability to get the state of your entity at any point in time, and building multiple views over the same data), it is by far more complex than CRUD services that update the state of an entity stored in a database.

Disadvantages of event sourcing include:

1. Complexity — In order to make sure read performance is not affected by a growing list of events you need to play, entity state snapshots have to be taken from time to time to reduce the performance penalty.
   This increases the system complexity with a background process that may have its own issues and when it does the data remain stale. On top of this, having 2 copies of the data means they might get out of sync.
2. Snowflake nature — Unlike CRUD ORM solutions, It’s harder to create common libraries and frameworks to ease development that can globally solve snapshotting and read-optimizations that can fit for every single use case.
3. Only supports eventual consistency (problematic for read-after-write use-cases)

Event Sourcing alternative — CRUD+CDC

Utilizing both simple CRUD capabilities and publishing database change events (CDC) for downstream uses (e.g. creation of query-optimized materialized views) can reduce complexity, increase flexibility and still allow for Command-Query Responsibility Segregation (CQRS) for specific use cases.

For most of the use-cases, the service can expose a simple read endpoint that will fetch the entity’s current state from the database. As scale increases and more complex queries are needed, the additional published change events can be used to create custom materialized views specifically tailored for complex queries.

In order to avoid exposing DB changes as a contract to other services, and creating a coupling between them, the service can consume the CDC topic and produce an “official” API of change events similar to the event stream created in event-sourcing pattern.

3. No Context Propagation

Switching to event driven architecture means developers, devops and SREs potentially have more difficulty to debug production issues and track the processing of end-user requests throughout the system.
Unlike with the request-reply model, there is no explicit chain of HTTP/RPC requests to follow. Debugging code is harder as event handling code is spread out across the service code instead of being sequentially traceable by clicking into function definitions usually found in the same object/module.

Consider for example the ecom flow that I’m using throughout this article. The Orders service has to consume multiple events from 3 different topics all related to the same user action (purchasing items in a webstore).

The other services also consume multiple events from one or more topics. Let’s assume that it was discovered that some inventory level is incorrect. Being able to investigate all related order handling events is crucial. Otherwise it will take a long time to go to individual services logs and try to manually connect the different pieces of evidence into one cohesive narrative.

Automatic Context Propagation

Automatically adding identification of the broader request context for all of the events, makes it really simple to filter for all events related to the end-user request. In our ecom example, 2 event headers were added — requestId and userId. Both of these IDs can greatly help with investigations.

At Wix, Greyhound automatically propagates the end-user request context when events are produced and consumed. In addition the request context is found also in logs infrastructure, such that logs can be filtered for specific user requests.

4. Publishing Events with Large Payloads

When processing large event payloads (payloads bigger than 5MB, e.g. Image Recognition, Video Analytics, etc…) it may be tempting to publish them to Kafka (or Pulsar) but there is a risk of greatly increasing latency, reducing throughput and increasing memory pressure (especially when tiered storage is not used)

Fortunately, there are a few ways to overcome this issue. Including introducing compression, splitting payloads to chunks, and putting the payload in object store and just passing a reference in the streaming platform.

Large Payloads Remedy I — Compression

Both Kafka and Pulsar allow compression of payloads. You can try several compression types (lz4, snappy, etc.) to find the one best suited for your payload type. If your payload is a bit large (up to 5MB), compression of 50% can help make sure you maintain good performance of your Message broker clusters.

Compression on Kafka level is usually better than application level, as payloads can be compressed in batches and thus improve compression ratio.

Large Payloads Remedy II — Chunking

Another way to reduce the pressure on brokers and override message size limitations is to split the messages into chunks.
While chunking is already a built-in feature of Pulsar (with some limitations), for Kafka chunking has to happen on the application level.

Examples of how to implement chunking on application level can be found here and here. The basic premise is for producers to send out the chunks with additional metadata that helps consumers to re-assemble them.

The two examples approaches are different in how they assemble the chunks back to the original payload. The first example keeps the chunks in some persistent storage and the consumer fetches them once all chunks have been produced. The second example makes the consumer seek backwards in the topic partition to the first chunk, once all chunks have arrived.

Large Payloads Remedy III — Reference to Object store

The final approach is to simply store the payload in an object store (such as S3) and pass a reference (a URL usually) to the object in the event payload. These object stores allow to persist any required size without impacting first byte latency.

It’s important to make sure the payload is fully uploaded to the object storage before the link is produced, or else the consumer will need to keep retrying until it can start downloading it.

5. Not handling duplicate events

Most message brokers and event streaming platforms guarantee at least once delivery by default. Meaning that some events are duplicated in the stream or potentially can be processed twice (or more).
The term for making sure a side-effect of a duplicate event only happens once is Idempotency.

Consider the simple ecom flow that I’ve been using throughout this article. In case of duplicate processing due to some processing error, inventory levels recorded into the Inventory database for purchased items may drop more than they actually should.

Other side-effects include calling a 3rd party api more than once (In our ecom case, it could mean calling inventory service with reduce-level twice for same event and item)

Idempotency Remedy — revisionId (versioning)

Optimistic locking technique can serve as inspiration in cases where idempotency of event processing is needed. With this technique, the current revisionId (or version) of the stored entity is first read before any update occurs. If more than one party tries to update the entity (while incrementing the version) at the same time (concurrently), the 2nd attempt will fail as the version will no longer match with what it read before.

In the case of idempotent handling of duplicate events, the revisionId has to be unique and part of the event itself in order to make sure that two events don’t share the same id and that 2nd updates on the same revisionId will (silently) fail even if does not happen concurrently.

Specifically With Kafka, there is a possibility to configure exactly once semantics, but still DB duplicate updates can happen due to some failure. Luckily the txnId in this case can just be the topic-partition-offset which is guaranteed to be unique.

* For more on Exactly once delivery in Kafka you can watch my talk from DevOpsDays Tel Aviv Conference

I would like to thank Oded Apel, Dalia Simons and Evgeny Krasik for their great feedback!

Summary

A migration to event-driven architecture can be gradual in order to reduce risks involved with it including harder debugging and mental complexity. Microservices architecture allows flexibility in the choice of pattern for each of the different services. HTTP/RPC endpoints can be called as part of an event’s processing, and vice versa.

As a consequence of this gradual migration approach I strongly recommend to adopt the CDC pattern (Database changes streamed as events) as a way to both ensure data consistency (pitfall #1) and avoid the complexity and risks associated with full blown event-sourcing (pitfall #2). The CDC pattern still allows to have the request-reply pattern in place side by side with the event processing pattern.

Fixing Pitfall #3 (propagation user request context throughout your event streaming flows) will greatly improve your ability to find root causes of production incidents quickly.

Remediations for Pitfalls #4 and #5 are for more specific uses cases — very large payloads in case of pitfall #4 and non-idempotent side-effects in case of #5. No need to perform the recommended changes in case there is no need for them. Although compression (#4) and transactionIds (#5) are best-practices that you can add by default.

Thank you for reading!

If you’d like to get updates on my future software engineering blog posts, follow me on Twitter and Medium.

You can also visit my website, where you will find my previous blog posts, talks I gave in conferences and open-source projects I’m involved with.

If anything is unclear or you want to point out something, please comment down below.

Further reading:

• 6 Event-Driven Architecture Patterns
• Troubleshooting Kafka for 2000 Microservices at Wix
• 4 Microservices Caching Patterns at Wix
• Leader election and sharding practices at Wix Microservices
• Gwen Shapira on Event Driven Architecture and Cloud Native Kafka

Event Driven Architecture — 5 Pitfalls to Avoid was originally published in Wix Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

In 2021, my team worked on migrating Wix’s 2000 microservices from self-hosted Kafka clusters to a multi-cluster Managed cloud platform (Confluent Cloud) in a seamless way that did not need to involve the owners of these services. The migration was done during regular traffic without any downtime.

This was the most challenging project I have ever led, so in this post I want to share with you key design decisions we took, best practices and tips for this kind of migration.

Split Overloaded Clusters

During the last couple of years, usage of Kafka by business OLTP services at Wix has increased dramatically, as more and more services incorporated event driven architecture.

Our self-hosted cluster scale increased from 5K topics with >45K partitions in 2019 to 20K topics with >200K partitions.

Traffic also increased from ~450M records produced per day to >2.5B produced per day per cluster.

This puts a big strain on the cluster controller startup time, as it has to load all this metadata on partitions, which makes the leader election duration greatly increase. It also has a big effect on startup time of individual brokers if they happen to go down causing more and more incidents of under-replicated partitions.

In order to avoid this instability in our Kafka clusters in production, we decided to migrate our self hosted Kafka clusters to Confluent Cloud and split the single-cluster-per-data-center to multiple clusters.

Why Managed Kafka

Self-managing a Kafka cluster is no easy feat, especially when performing tasks such as rebalancing partitions among brokers, or upgrading broker versions. These can be quite scary and require careful planning and execution. That is especially true when the clusters become overloaded with metadata as was our case.

Below are 4 benefits of using a managed Kafka cloud platform and Confluent Cloud in particular:

1. Better Cluster performance & flexibility
   Brokers Partition rebalance is done for you, so brokers don’t become performance bottlenecks. You can also easily scale the cluster capacity up or down in order to achieve cost effectiveness.
2. Transparent version upgrade
   The Kafka code base is being improved all of the time, especially focused on KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum. Which, when completed will dramatically increase the amount of metadata each cluster and broker can hold. By going managed, you get automatic version upgrades which improve performance and fix bugs.
3. Easy to add a new Cluster
   In case you require a new cluster. Setting it up is dead simple.
4. Tiered Storage
   Confluent Platform offers Tiered Storage, which allows to dramatically increase the retention period for Kafka records without paying for expensive disk space, by shifting older records to much cheaper S3 storage, without adding latency to latest-offset record consumption.

Switching 2000 Microservice to Multi-Cluster Kafka architecture

At Wix we have a standard JVM library and Proxy service for interacting with Kafka called Greyhound. It’s an open source high level SDK for Apache Kafka which offers additional features such as concurrent message processing, batch processing, retries, and more. You can read more about it in this article or you can watch this talk.

Greyhound is the event-driven backbone of Wix ~2200 microservices, so introducing the multi-cluster concept only had to be done in a few places (including the library and the proxy service). Only new topics’ producers and consumers have to explicitly specify the cluster, as old ones are automatically mapped.

All that is needed by developers is to define the logical name of the cluster when constructing the greyhound consumer or producer, as can be seen below:

How to Split

We decided to split the Kafka clusters according to various SLAs. For example, CI/CD pipelines and data migration use cases have different SLA than production services. Important to note that not all data centers have the same amount of Kafka clusters, as each data center serves a different purpose.

Greyhound (Wix’s own Kafka SDK) knows how to address this issue and avoid crashing in case a cluster is not available in the data center where the service instance is currently running.

Migrate Traffic Drained Data Center?

In order to make it easier to migrate producers and consumers of 2000 microservices to multiple managed Kafka clusters, the initial design relied on first completely draining each data center (DC) from traffic.

This design meant that all that was required was to switch the connection details of producers and consumers to their new Kafka clusters. As Wix microservices used Greyhound layer to connect to Kafka clusters, changing connection was required in only one place — Greyhound production configuration (while making sure only 1 DC is affected).

As simple as this design was, it was unfortunately not possible to implement and execute.
There were several reasons:

1. There are services that are only deployed in one of the data centers and it is difficult to move them.
2. This design means all-or-nothing and is very risky for the moment traffic returns. In case of edge cases that were not switched, they may suffer data loss.
3. Traffic cannot be completely drained from a data center for a long period of time, because that could greatly increase the risk of downtime to some service.

Instead, a new design was planned involving migration during live traffic.

Migrate With Traffic

Performing migration during live traffic meant that meticulous planning and execution had to take place.

The process had to be gradual (only affecting some microservices each time in order to reduce blast radius in case there is an issue) and completely automated, in order to reduce chance of manual errors, including automation of a rollback process.

Migrating producers first (before consumers) was not an option, as it meant a considerable amount of time was needed to make sure all consumers have finished processing all records found on the self-hosted cluster and can safely switch to the new cluster topic. This would have caused considerable delays in processing which could harm certain OLTP business flows and users. Also, rolling back consumers due to some unexpected issue would have been impossible without data loss.

Active Kafka consumers had to be switched first, while making sure no message is lost and there is minimal re-processing of records. The only way to do this would be to first replicate the records of all the consumed topics from self host cluster to destination managed cluster.

Replication

In order to make sure no message processing is lost during migration, we created a dedicated replication service. Once all consumer topics were identified, the replicator service was ordered to create the topics in the appropriate cloud clusters, and start consuming records from the self hosted cluster and produce them to the target cluster.

Consumer Migration

In order to facilitate consumer migration, the Replicator also persisted the offset mapping for each partition such that the Greyhound consumer can start processing records in the cloud cluster from the correct offset — which was mapped from the first non committed offset in the self-hosted cluster.

In addition to the Replicator, there was a Migration Orchestrator that made sure a topic was replicated, all offsets were mapped, and then proceeded to request the Greyhound consumer to unsubscribe from the self-hosted cluster. After verifying this was successful, the Orchestrator requested the consumer to subscribe to the cloud cluster, while first seeking the correct mapped offset.

In case of failure, the Orchestrator was able to request the consumer to revert back to the self-hosted cluster.

All this Orchestrator to consumer communication was itself done via dedicated Kafka migration topics. Greyhound consumers started listening to them when they were started.

Producer Migration

Once all consumers of a certain topic were migrated, it was possible to migrate its producer.
The initial migration design entailed requesting the producer to switch cluster connection while still accepting incoming produce requests. This meant buffering the requests in-memory and was deemed quite risky.

We later came up with a simpler design which relied on Wix’s gradual Kubernetes deployment process. Each new pod starts to accept incoming requests only once all its health checks are OK, including connection to Kafka. As this process is gradual, there are always “older” pods that are running such that the service as a whole is always available for incoming requests.

On pod startup, the Greyhound producers simply checked a database to ascertain to which cluster they should connect. Much simpler then dynamic cluster switching and records buffering. This meant the migration was safe, no requests could be lost and the service was kept at high availability.

Replication bottleneck

Only once a producer was migrated, was it acceptable to stop a topic’s replication. But in order to migrate a producer, first all its topic consumers had to be migrated.

It turned out that many topics had multiple consumers from different services, which meant the Replicator consumer had more and more traffic to process and replicate.

This caused an issue when a limit was reached to the amount of topics the consumer could handle due to a technical limitation with our relatively old version of self-hosted Kafka brokers. After several attempts to increase message.max.bytes backfired (see KAFKA-9254 bug) and caused a serious issue, we decided to simply add more Replicator consumers and shard the topics-to-replicate among them.

Beyond Migration — External Consumer Control

This “with traffic” migration design opened many new possibilities to dynamically change Greyhound consumers configuration or state without needing to GA a new version to production.

As now we have the infrastructure in place for Greyhound consumers to listen to incoming commands to change state or configuration. Such commands can include:

1. Switch cluster — unsubscribe from current cluster and subscribe to another one
2. Skip records — skip records that can’t be processed
3. Change processing rate — dynamically increase or decrease amount of parallel processing or add delay for throttling and back-pressure
4. Redistribute records — In case there is a growing lag on a single partition, be able to redistribute records among all partitions (and skip the old ones).

Currently these are all theoretical, but can be much more easily implemented using the migration infrastructure already in place.

Best Practices and Tips

Below is a list of best practices and tips for a successful Kafka clusters migration:

1. Create a script that checks state by itself and stops if expected state is not reached
   Making the migration process as automatic as possible is key, so having the script check by itself if it can move to the next phase greatly expedite the process. Un the other hand auto-rollback and self-healing is very difficult to get right, so best to leave that for manual intervention.
2. Have a rollback readily available
   No matter how well you tested your migration code, a production environment is the realm of uncertainty. Having a readily available rollback option for each phase is very important. Make sure to prepare it in advance and to test it as much as possible before you start running the migration.
3. Start with test/relay topics and no impact topics
   Migration can be very dangerous, as records can potentially be lost or recovery can be painful. Make sure to start testing your migration code with test topics. In order to have a realistic examination. Use test topics that actually mimic production topics by replicating real production records to the test topics in a special test app. This way, failure in the consumers migration process won’t have any production affect, but it will provide you with a more realistic production simulation.
4. Create custom metrics dashboards that show current and evolving state
   Even if you create a fully unattended automatic migration process, you still need to be able to monitor what is going on and have tools to investigate in case there is an issue. Make sure to prepare dedicated monitoring dashboards in advance that clearly show the current and historical state of consumers and producers you are migrating.
   In the graph below we can see how a producer successfully switched from self hosted cluster to managed cluster (throughput goes down as more and more pods are restarted and read the new configuration)
5. Make sure to be on latest patch version of your self-hosted Kafka broker
   As our self-hosted Kafka brokers were not on latest patch version, we ended up with a production incident when trying to increase the value of message.max.bytes several times (see more in Replication bottleneck section of this post). My recommendation is to first increase your self-hosted cluster Kafka brokers versions. If not to latest, then at least to latest patch version.

Summary

We used Greyhound & dedicated orchestration services and scripts in order to achieve an automatic, safe, and gradual migration in a seamless manner during live traffic.

This was not an easy task. In case you can completely drain traffic from your data-centers and/or afford processing down time, switching producers and consumers to their new clusters without replicating the data first is highly recommended. It’s a much simpler design that will take you much less time to get done.

Otherwise, when migration under traffic, you need to keep careful attention to the order in which you perform the migration (Consumers before/after Producers) and make sure you understand the consequences of this decision (ability to rollback, chance of losing data).

Below you will find a handy flowchart to make it easy for you to understand the various options.

Thank you for reading!

If you’d like to get updates on my future software engineering blog posts, follow me on Twitter and Medium.

You can also visit my website, where you will find my previous blog posts, talks I gave in conferences and open-source projects I’m involved with.

If anything is unclear or you want to point out something, please comment down below.

Further reading:

• Event Driven Architecture — 5 Pitfalls to Avoid
• 6 Event-Driven Architecture Patterns
• Troubleshooting Kafka for 2000 Microservices at Wix
• 4 Microservices Caching Patterns at Wix
• Leader election and sharding practices at Wix Microservices
• Gwen Shapira on Event Driven Architecture and Cloud Native Kafka

Migrating to a Multi-Cluster Managed Kafka with 0 Downtime was originally published in Wix Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

If you realize down the line that you’ve made the wrong choice, migrating to another database is a very costly and risky procedure.

Each database technology (and type) has advantages and disadvantages.

While there is a convergence happening in some general-purpose databases (that support various models relational / document / key-value at the same time) like MySQL and PostgreSQL, which claim to make your life easier,

Cloud providers like Amazon offer a dazzling array of database and storage options (e.g. Amazon has Aurora, RDS, DynamoDB, DocumentDB, Keyspaces, elastiCache, S3, elastic file system, etc…), making it harder to figure out which one is the right one.

Consider the following criteria for choosing the right database technology for your service:

Query Patterns

How complex are your query patterns? Do you just need retrieval by key, or also by various other parameters? Do you also need fuzzy search on the data?

How you are going to fetch your data is one of the main ways to find the best database for your use case. If you are going to fetch data by key, then all you need is a key-value store (e.g. DynamoDB, Redis, S3, GCS).

In case you mostly fetch via key, but sometimes you also need to fetch by one or two other fields, then Wide-Column DBs (e.g.DynamoDB, Cassandra) may be right for you.

On the other hand if you will require to query by many different fields you can choose either Relational DB (e.g.MySQL, PostgreSQL) or Document DB (e.g.MongoDB, CouchDB, MySQL, PostgreSQ). Note that Document DBs don’t support well queries that require joining data from multiple documents.

Finally, in case you are looking for fuzzy search query capabilities (free text search), then search engines like Elasticsearch and Solr are the best fit.

Consistency

Is strong consistency required (read after write, especially when you switch writes to a different data-center) or eventual consistency is OK?

In case you need to read your data right after your write it (i.e. strong consistency) than a Relational database (e.g. MySQL, PostgreSQL) is usually more suited than a Document Database (e.g.MongoDB, CouchDB), especially in case of multi-data-center scenario.

Storage Capacity

How much storage capacity is needed?

Most database systems are limited by the amount of space on disk (e.g. MySQL) or struggle with performance as amount of Nodes and Shards grows into the hundreds (e.g. Elasticsearch).

When infinite storage is needed this is where cloud solutions shine. Object Storage Services like S3 and GCS will allow you to store as much data as you like with the handy option is multiple tiers, so you pay less for data that is rarely retrieved.

Performance

What is the needed throughput and latency?

All databases performance degrades as the amount of read/write throughput traffic increases. This is the time when optimizations such as re-indexing and re-sharding of your data come in handy.

In case you have very high traffic and require very low latency, Cloud providers solutions like Amazon’s DynamoDB and Google’s Bigtable could be just what you need. As long as your service is deployed on the same data center as the database, you can enjoy latencies that are under 10ms. The downside is of-course the $ cost.

Maturity and Stability

If you choose self-hosted deployment, How much experience does your DBA team have with this technology, how mature is it?

Choosing the most trendy, powerful, and fully featured database to self-host maybe tempting, but as long as your orgainization doesn’t have experience with this database, you may end up regretting it.

Setup, configuration and fine tuning of databases is a lengthy and risky ordeal. Sometimes choosing the “old” organization self-hosted work-horse will pay bigger dividends in the long term when it comes to production stability.

Cost

If you choose a managed cloud solution, What are the costs? What are its limitations?

The payment model for managed cloud solutions is usually proportional to the read/write traffic. Make sure to read the finer print for each managed solution and make sure that it is cost-effective for your specific read/write usage patterns.

Below you will find a handy flowchart to help you choose the right database technology for your needs.

Recently, I wrote an article about 5 Database technologies used by 2000 Wix microservices, where I discuss the pros and cons of each technology and various specific Wix use cases for each one. The technologies are MySQL, MongoDB, DynamoDB, Elasticsearch and S3.

Thank you for reading!

If you’d like to get updates on my future software engineering blog posts, follow me on Twitter and Medium.

You can also visit my website, where you will find my previous blog posts, talks I gave in conferences and open-source projects I’m involved with.

If anything is unclear or you want to point out something, please comment down below.

Further reading:

• 5 Database technologies used by 2000 Wix microservices
• Event Driven Architecture — 5 Pitfalls to Avoid
• 4 Microservices Caching Patterns at Wix
• 6 Event-Driven Architecture Patterns

How to choose the right database for your service was originally published in Wix Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Unlike with UI frameworks, or software industry buzzwords, data storage technologies, i.e. databases are famously slow to change, and new innovation takes longer to be adopted.

For a large organization, switching to a different OLTP database as your main workhorse happens very rarely. In recent years, with the advent of microservices, more flexibility is given for each service to choose the way it persists it’s data. Still, the risk of using unproven technology sometimes outweighs the benefits, as losing crucial business related data is unacceptable. While the technology may be proven elsewhere, it may not be proven and battle-tested in your organization yet.

At Wix, the situation is not different, as MySQL is still the main go-to database, more than 10 years after it was first used. Albeit, new and modern MySQL APIs (e.g. JSON Documents) are being used more and more.

But with 2000 microservices in production, there are of course several storage technologies other than MySQL that are used by a significant portion of services due to various reasons. The reasons range from different API use-cases (e.g. support CRUD API, allow free-text search) to different tech stacks (JVM vs. Node.js) to more demanding technical requirements (infinitely scaling storage).

5 technologies are described below, each with its own use cases at Wix, unique characteristics, strengths and weaknesses. The technologies are MySQL, MongoDB, DynamoDB, Elasticsearch and S3.

Read on to get some insights that may help you choose your next microservice storage technology.

1. MySQL

Other DBs in this category
PostgreSQL, MariaDB, SQL Server, Oracle, Db2, SQLite

When to use MySQL

• When you already widely use it in your organization
• When you want both relational tables (when you know the schema upfront) and JSON collections (Schemaless)
• Relational / normalized — when you need to optimize on writes instead of reads, to have strong read consistency

MySQL Advantages

• Maturity & Reliability — MySQL is highly used, battle tested and mature
• Fast read performance
• Improved JSON/Document support (MySQL 8)
• Cross-DC write consistency (when ProxySQL is used)

MySQL Disadvantages

• Scalability — Does not scale horizontally. Limited by amount of disk space
• Consistency and Replication Issues (when not using ProxySQL)

Example MySQL use cases at Wix

A. Pre MySQL8 JSON support — relational tables with BLOB
Almost all JVM (Scala/Java) services at Wix used to store their data in MySQL relational tables. Data was usually denormalized across multiple tables (SQL joins were rarely used). For added flexibility most of the content was kept in BLOB type in order to avoid frequent schema changes that add a lot of risk for production stability.

B. JSON support and Simple Data Layer
Simple Data Layer (SDL) is an internal infrastructure built at Wix in order to help speed-up the development time and reduce cognitive load when persisting a document data structure.

This is a very common pattern as persistence exists in CRUD methods as well as business operations. The value spans from having sensible defaults like page size through support for WixQL which is part of our API and up to wider cross cutting concerns like GDPR support, emitting domain-changed events, and more.

It supports the very common need of microservices for a non relational embedded document data structure (to have more operations done by DB instead of in-memory).

Its engine is MySQL8 with JSON datatype. MySQL was chosen mainly because of Wix DBAs long time experience with MySQL, a lot of infrastructure already in place for backups, migrations, indexing, and for the strong consistency and availability support provided by ProxySQL (see next paragraph)

C. Multi DC & replication support with ProxySQL
ProxySQL is used in order to seamlessly route write traffic to the currently active main data center’s MySQL clusters without any data loss. It also manages read distribution among replicas. It provides high availability and high performance out of the box.

2. MongoDB

Other DBs in this category
CouchDB (many relational/NoSQL DBs now offer JSON support)

When to use MongoDB

• When data schema is predicted to keep changing and evolving
• When working with dynamic JSON content
• When keeping data denormalized is not a problem. i.e. to have eventual consistency

MongoDB Advantages

• Flexibility — with Schemaless documents, the number of fields, content and size of the document can differ from one document to another in the same collection.
• Easy to scale with sharding

MongoDB Disadvantages

• High Memory Usage — a lot of denormalized data is kept in memory
• Document size limit — 16MB
• Non optimal replication solution (data cannot be re-replicated after recovery from failure). Consistency issues on traffic switch to another data center (No automatic remaster)

Example MongoDB use cases at Wix

A. Wix Data
Wix Data is a service that allows anyone with a Wix website to own a document database — “Database as a Service”). It uses MongoDB behind the scenes to store the data.

Mongo was chosen in order to be able to store any schemaless (JSON) doc. When the team compared MongoDB performance to MySql 5.7.6, the former had 2X write and read performance over the latter (albeit, this older version of MySql didn’t support JSON natively yet)

Recently the Wix Data team published a post on how to host 1 million sites per 1 MongoDB cluster

B. Wix Blog
Wix Blog is Wix’s blog publishing site-app. The backend Node.js microservices use MongoDB to store several TBs worth of blog posts.

When the project was young the schema was still evolving, and MongoDB’s schema-less nature helped ease the pain of adding new fields to a fast growing database. Data is mostly kept denormalized as MongoDB joins ($lookup) are not convenient to use.

3. Amazon DynamoDB

Other DBs in this category
Cassandra, HBase, Redis (Key-value Store)

When to use DynamoDB

• When you need a simple key value store without complex querying patterns
• When you need to store expirable data
• Low-medium throughput apps as writes are expensive and consistent reads are twice the cost of eventually consistent reads.

DynamoDB Advantages

• Fast performance in any scale (as long as enough capacity is provisioned)
• No storage limit
• Schemaless — it’s possible to define a schema for each item, rather than for the whole table
• Multi-master replication (update data in multiple regions)
• Supports TTL per item
• Built-in CDC events (DynamoDB streams)

DynamoDB Disadvantages

• Size limit — item can only reach 400KB in size
• Limited querying options (limited number of indices)
• Throttling on burst throughput (and hot keys in certain situations)

Example DynamoDB use cases at Wix

A. KVStore + Cache
DynamoDB is the back-bone for Wix KV-store cache abstraction. You can find more details in this article (search for (Dynamo)DB+CDC based Cache)

B. Wix Chat
Wix Chat is a service that lets site owners instantly communicate with visitors.

They recently migrated their database from Cassandra to DynamoDB, in order to be able to handle growing scale without needing to increase the DBA & operations teams.

In DynamoDB the Wix Chat team stores chat messages and chat message indications (sent / seen / failed). In addition, data is stored about online site visitors (page, location, entrance time, etc.) which also takes advantage of DynamoDB TTL feature.

C. Wix Contacts
Wix Contacts is a service that manages all contacts of a Wix site owner (Part of Ascend business tools suite)

DynamoDB is used to help orchestrate event-driven bulk jobs (they ​​save the job current status and current offset/page). Each of the job stages is advanced by sending a Kafka event to the appropriate topic. Then the Relevant data is fetched from DynamoDB.

4. Elasticsearch

Other DBs in this category
Apache Solr, Splunk, Amazon CloudSearch

When to use Elasticsearch

• When you need to perform fuzzy search or have results with ranking
• When you have another data store as source of truth (populate Elasticsearch as a materialized view)

Elasticsearch Advantages

• Easy to horizontally scale with index sharding
• Rich search API
• Query for analytical data using aggregations

Elasticsearch Disadvantages

• Indexes are created with a predefined number of shards. More shards requires migration to a new index. Usually done with ReIndex API
• Performance issues when indices hit very large scale (> 1TB with hundreds of nodes and shards)

Example Elasticsearch use cases at Wix

Wix Search
Wix search allows searching inside any Wix site (out of the box) by having a generic API for indexing per site and installed apps.

It also provides custom search services for creating and managing self-owned indices by other Wix teams (basically serves as a proxy to Elasticsearch).

5. Amazon Simple Storage Service (S3)

Other DBs in this category
Google Cloud Storage, Azure Blob Storage

When to use S3

• When you need to store large binary objects/files (up to 5TB each)
• When the amount of data you need to store is large (>10TB), continues to grow daily, and may need to be retrieved (can’t be deleted)

S3 Advantages

• Supports very high throughput
• Infinite scalability — No limit on amount of storage

S3 Disadvantages

• No Query support, only key-based retrieval
• Latency is 100–200 ms for small objects. Caching can alleviate this

Example S3 use cases at Wix

A. Wix Media
Wix Media manages all the media files that are uploaded by Wix site owners. These files are stored, and then accessed when the user’s site gets a visitor.

In order to have highest possible availability and lowest possible retrieval latency, the media files are stored and replicated between both Amazon S3 and Google GCS and cross-region within the same cloud provider.

A dedicated process moves files to cheaper storage classes (e.g. from S3 Standard to S3 Standard-IA) once it finds out they haven’t been accessed for a long time. for more information you can read — Storage Lifecycle: How We Cut Cost on 55% of Our Storage Without Deleting a Single File

B. Wix Editor
The Wix Editor backend team is in charge of (among other responsibilities) storage and fast retrieval of all Wix site pages during site editing.

A site page is actually a JSON file. Every change made to a site page means a new immutable JSON file is inserted to the database.

A couple of years ago they migrated the site pages from MySQL to S3. The main reason was that S3 guaranteed infinite scale (MySQL was unable to handle a scale >10TB well). The S3 latency they measured for read/write ranged from 30 ms on median case to 400ms on p99. In order to improve this they added a write-through caching layer on top of S3.

Summary

Each database technology (and type) has advantages and disadvantages.

While there is a convergence happening in some general-purpose databases (that support various models relational / document / key-value at the same time) like MySQL and PostgreSQL,

Cloud providers like Amazon offer a dazzling array of database and storage options (e.g. Amazon has Aurora, RDS, DynamoDB, DocumentDB, Keyspaces, elastiCache, S3, elastic file system, etc…)

When choosing a database technology it is very important to consider various criteria:

1. How complex are your query patterns? Do you just need retrieval by key, or also by various other parameters? Do you also need fuzzy search on the data?
2. Is strong consistency required (read after write, especially when you switch writes to a different data-center) or eventual consistency is OK?
3. How much storage capacity is needed?
4. What is the needed throughput and latency?
5. If you choose self-hosted deployment, How much experience does your DBA team have with this technology, how mature is it?
6. If you choose a managed cloud solution, What are the costs? What are its limitations?

Below you will find a handy flowchart to help you choose the right database technology for your needs.

For more in-depth look at the various criteria you can read my follow-up article — How to choose the right database for your service

I would like to thank Giedrius Graževičius, Karolis Astrauka, Donatas Remeika, Ittai Zeidman, Baruch Assif Osoveskiy, David Vaks, Avraham Rosenzweig, Idan Yael, & Eyal Malron for their help in detailing their DBs use-cases and technical insights

Thank you for reading!

If you’d like to get updates on my future software engineering blog posts, follow me on Twitter and Medium.

You can also visit my website, where you will find my previous blog posts, talks I gave in conferences and open-source projects I’m involved with.

If anything is unclear or you want to point out something, please comment down below.

Further reading:

• How to choose the right database for your service
• 4 Microservices Caching Patterns at Wix
• Event Driven Architecture — 5 Pitfalls to Avoid
• 6 Event-Driven Architecture Patterns

5 Database technologies used by 2000 Wix microservices was originally published in Wix Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Fibers are the backbone of the highly performant, asynchronous and concurrent abilities of ZIO. They are lightweight “green threads” implemented by the ZIO runtime system.

Forking any IO[E, A] effect means it will immediately run on a new fiber.

The ZIO documentation recommends that non-advanced programmers avoid fibers and instead use concurrent primitives like raceWith, zipPar, foreachPar, and so forth, which is sound advice.

But very soon in a lot of different applications, there comes a need to fork a new fiber. The best example would be — running some recurring job.

Fiber management edge cases
When managing the application’s fibers there are inevitable edge cases each ZIO user has to be familiar with and be comfortable with.

Errors propagate much better in the asynchronous ZIO runtime (via tracing) than within the Scala Future mechanism. But there are still cases where an error can cause a fiber to die unexpectedly.

Fibers can also get stuck on some blocking IO operation, and while it may be interruptible, the fact that the fiber is stuck may not be obvious as no error is thrown.

Track, debug and avoid
This article is about how to track, debug and avoid unexpected behaviour in ZIO fibers, including:

1. Making sure that fibers do not die without at least some kind of reporting
2. Making sure execution will not get stuck when interrupting a fiber
3. Having easily available fiber tracking (fiber dump) in place through ZMX or a custom solution

1. Handling unexpected failures that can cause fibers to die

The code of the example application shown below, creates a periodic job (using the repeat operator) that is forked to run on its own fiber.

In order to make sure the job continues to run periodically even when an error occurs the catchAll operator is used before the repeat operator.

catchAll doesn’t catch all failures
Unfortunately, the catchAll operator only catches expected failures. E.g. throwable error on side-effects imported by ZIO.effect or pure errors produced by ZIO.fail

Unexpected failures (aka defects), such as an exception thrown in pure effect ZIO.succeedare not caught by catchAll and will cause the fiber to die.

This is demonstrated in the standard output of the application:

do other stuff
Fiber failed.
An unchecked error was produced.
java.lang.RuntimeException: unexpected defect
…
do other stuff
do other stuff
do other stuff
…

(Note that tapError & retry operators also don’t catch such failures).

catchAllCause does catch all failures
The E type of ZIO[R, E, A] refers only to expected failures. Cause[E] is a description of a full story of failure. Cause can be either Fail[+E](value: E) or Die(value: Throwable)

Changing catchAll to catchAllCause will make sure the fiber continues even in case of unexpected error.

This is reflected in the output of the altered program:

do other stuff
job failed with Traced(Die(java.lang.RuntimeException: unexpected defect)…
do other stuff
job failed with Traced(Die(java.lang.RuntimeException: unexpected defect)…
…

(Note that you can use cause.squashTrace that squashes a Cause down to a single Throwable)

Usually it is not recommended to keep the fiber going in such extreme cases, but you may require otherwise in your use-case.

tapCause to help with debugging
tapCause is not able to keep the fiber going, but it will at least allow the unexpected error to be reported to the log with all additional information to help with investigation:

do other stuff
job failed with Traced(Die(java.lang.RuntimeException: unexpected defect)…). extra information…
Fiber failed.
An unchecked error was produced.
…

withReportFailure
You can also set your Runtime Platform’s .withReportFailure in order to report on any uncaught errors, including defects.

runtime.mapPlatform(_.withReportFailure(_ => [report unexpected failure]))

2. Interrupt a Fiber after a grace period

Unlike with Scala Futures, ZIO fibers can almost always be interrupted. Only if uninterruptible is specified explicitly, or inside managed resource acquire effect, will the fiber not be interrupted.

There are certain situations where interrupting a fiber is required. For example, when releasing a managed resource that forked a fiber when it was acquired.

Abrupt interruption
In the following example, a long running fiber is abruptly interrupted after a while.

The application output shows how the fiber termination happened, immediately after the interrupt effect was interpreted:

19:15:38.445 long running job…
19:15:38.501 do stuff
19:15:39.696 do stuff
19:15:40.701 do stuff
19:15:40.862 finalizing job!
19:15:40.871 do other stuff
19:15:41.873 do other stuff

The ensuring operator allows to run any finalization effect that is required at the end of fiber execution (in this case it helps with report of time of interruption)

Graceful interruption
In case it is beneficial to allow the fiber a grace period before interruption, the code can be changed as follows (see line 7):

1. fiber.join — in order to set a timeout, the fiber has to be joined
2. .resurrect.ignore — resurrect converts unexpected failures (defects) to regular errors in order to make sure all types of errors are ignored. Otherwise any unexpected failure in the forked fiber will cause the main fiber to die as well when the former is joined.
3. The timeout is specified together with disconnect in order to make sure that timeout will not be blocked by an uninterruptible effect (e.g. due to releasing a managed resource for instance). Setting the job as interruptible also makes sure the timeout will not be blocked

Line 8:

1. timeout.isEmpty — If the timeout elapses then None is returned.
2. In the case of a timeout the fiber is still not interrupted, so an explicit interrupt is needed. Here interruptFork is used in case the forked fiber was marked as uninterruptible in order to avoid waiting forever for an uninterruptible to get interrupted.

Now, the output is as follows:

19:23:01.422 long running job…
19:23:01.492 do stuff
19:23:02.689 do stuff
19:23:03.693 do stuff
19:23:08.960 finalizing job!
19:23:08.963 do other stuff
19:23:09.964 do other stuff

As the output shows, the fiber was only interrupted after 5 seconds (the timeout period)

Important note: marking an effect to be forked as interruptible will not have meaning in case the effect itself is uninterruptible. interruptible is only helpful inside some part of a bigger uninterruptible effect.

3. Monitoring Fibers state

Leaking fibers can cause instability in any ZIO application. For example, performing too many fork operations on the zio Blocking thread pool (which has dynamic size) e.g. effectBlocking([some blocking effect]).forkDaemon can cause memory overloading due to too many open threads.

Fibers can also die suddenly (as we saw in the first part of this article on defects) or become stuck.

ZMX fiber dump
Performing a fiber dump is sometimes essential to debug the current (troubled) state of your application through the state of its fibers.

Fortunately, there is ongoing work to introduce such an ability to ZIO through the ZIO-ZMX project.

On the application (server) side, the requirement is to setup a diagnostics layer

val diagnosticsLayer: ZLayer[ZEnv, Throwable, Diagnostics] = Diagnostics.live(“localhost”, 1111)

And to supply the runtime Platform with a ZMXSuprvisor:

val runtime: Runtime[ZEnv] = Runtime.default.mapPlatform(_.withSupervisor(ZMXSupervisor))

Supervisors allow the current state of fibers to be tracked.

On the client side, the interactions is through TPC directly

echo -ne ‘*1\r\n$4\r\ndump\r\n’ | nc localhost 1111

Or through a Scala ZMXClient.

For more information, visit the Diagnostics documentation page of ZMX

Roll your own fiber dump
ZMX is not yet officially released (as of May 2021), and regardless you may wish to create your own custom way of accessing a fiber dump.

In order to implement your own fiber dump mechanism, follow these steps:

1. Create a FiberTracking Service that will access the supervisor’s fibers

2. Create a Runtime extension method that creates a supervisor with fiber tracking enabled (Supervisor.track(weak = false)), maps the runtime Platform to include this supervisor, and includes the FiberTracking service in the runtime environment.

3. Create a runtime for your application’s code that sets fiber tracking on using the extension method from the previous phase.

4. Upon demand, retrieve a snapshot of the currently running fibers from the FiberTracking service and secure a fiber dump string to do with as you wish.

Note: This example includes a simple console application, but the exact same code can be used for a web server as well.

Summary

This article described ZIO fiber related edge cases such as a fiber dying due to unexpected failure, or the requirement to interrupt a fiber, either abruptly or gracefully in a successful manner.

It recommended to use such ZIO operators as ensuring, tapCause, catchAllCause, withReportFailure, resurrect, and disconnect in or to make sure unexpected errors are caught (or at least reported), resources are then released and also that fibers are interrupted safely.

It also offered a detailed description of how to set up fiber tracking and executing a fiberDump for increased debuggability of your ZIO application.

I would like to thank Noam Berman, Dmitry Karlinsky and Asaf Jaffi for their help in making this article.

Thank you for reading!

If you’d like to get updates on my future ZIO related blog posts, follow me on Twitter and Medium.

You can also visit my website, where you will find my previous blog posts, talks I gave in conferences and open-source projects I’m involved with.

If anything is unclear or you want to point out something, please comment down below.

Further reading:

• 5 pitfalls to avoid when starting to work with ZIO
• 5 (more) pitfalls to avoid when starting to work with ZIO
• 5 lessons learned from my continuing awesome journey with ZIO
• Easily Create New Sync Types with ZIO STM

How to successfully manage a ZIO Fiber’s lifecycle was originally published in Wix Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.