Messaging: Carousell’s Asynchronous Communication Platform

How we built an in-house platform that connects 120+ microservices and handles around 5.5 billion messages every month.

Co-Authored with: Rajath Ramesh

Background

In late 2016, Carousell began moving from a monolithic backend to a microservice architecture. As the number of services grew, a simple problem became increasingly difficult: how should services communicate without depending too much on one another?

For example, when a listing changes, several systems may need to react to it. Having the Listing Service call each system directly would make it responsible for knowing about every consumer, making the setup harder to extend and maintain.

We needed a simple way for services to communicate asynchronously. That led us to build Messaging, Carousell’s in-house asynchronous communication platform.

Goals

Before building Messaging, we defined five key goals:

1. Support different communication needs: Pub/Sub (one-to-many), Async API (one-to-one), and Scheduled API (deferred one-to-one).
2. Keep infrastructure ownership simple: Messaging owns and manages the shared infrastructure, while each service only needs to manage its own logic.
3. Handle failed messages safely: Messages that repeatedly fail should be moved aside for investigation and replay, rather than retried forever or lost.
4. Make problems easy to investigate: Engineers should be able to follow a request across services and quickly understand where something went wrong.
5. Make onboarding self-serve: Teams should be able to add new publishers and subscribers without needing help from the platform team.

Platform Architecture

The platform is deployed on Kubernetes, except for the message broker and datastore. All components communicate through gRPC. Messaging does not need to understand the business payload it carries. The payload is wrapped in a Protobuf envelope, while its message type is agreed upon separately between producers and consumers through a central Schema Registry.

Component roles:

1. Functional RPC: The single gRPC entry point for all message sends, async calls, and schedule requests from microservices.
2. Management RPC: API for registering publishers, subscriptions, and async call callers — platform team and automation only.
3. Coordinator: Reacts to registration events: creates Kafka topics, deploys Handler pods, keeps state in sync.
4. Apache Kafka: Durable message broker; doubles as the sideline (DLQ) topic store.
5. PostgreSQL: Stores registration metadata and scheduled-call payloads.
6. Handlers: Auto-deployed K8s pods (one per subscription / async-call registration); scale independently based on volume.
7. Scheduler: Polls the schedule store by time shards and enqueues payloads at the right moment.

Use Case 1: Pub/Sub

The problem

In a marketplace, a listing change is relevant to many downstream systems. Search needs to reindex it. Promotions need to check eligibility. Quota needs to decrement limits. If the Listing Service calls each of them directly, it becomes responsible for every new consumer — brittle, and hard to scale.

How it works

The publisher registers once with a unique publisher_id. The Coordinator creates a Kafka topic for it. Subscribers register independently, providing their gRPC endpoint and optional message-type filters. The Coordinator deploys a Handler pod per subscriber that consumes from the topic and calls the subscriber’s endpoint.

Setup (one-time, via Management API):

// Register publisher
{ "publisher_id": "listingsvc", "description": "Listing lifecycle events" }

// Register a subscriber — Coordinator auto-deploys the Handler pod
{
  "subscriber_name": "search-ingest",
  "publisher_id": "listingsvc",
  "filter": { "message_types": ["listing_created", "listing_updated", "listing_deleted"] },
  "num_workers": 1,
  "host": "search-ingest-svc",
  "port": 9999,
  "endpoint": "/disocvery-ingestion/ReceiveMessage",
  "num_retries": 3,
  "timeout_ms": 1000
}

In production today, Listing service has over a dozen active subscribers — bump, search ingestion, promotions, offer service, and more — all added independently without any change to the publisher.

Use Case 2: Async API

The problem

When a user bumps a listing, several actions must happen: charge coins, update listing visibility, trigger marketing enrichment. Not all of them need to block the user’s response. Worse, if one downstream call fails, failing the entire user request and asking them to retry is a poor experience.

How it works

The caller registers as an async_call_caller and provides a list of callees with their gRPC endpoints. On AsyncCall, the Functional RPC wraps the request in an envelope (preserving the distributed trace context), publishes to a Kafka topic, and returns immediately. A Handler pod consumes from that topic and makes the actual gRPC call to the callee. If the call fails after configured retries, the message is moved to a sideline topic for inspection and manual replay.

Key design points:

• At-least-once delivery with configurable retries before sidelining
• Distributed trace propagation: the trace ID is extracted from the incoming gRPC context, serialised into the Kafka message envelope, and re-injected when the Handler makes the downstream call — giving a single trace across sync and async hops
• Optional callback: the callee can invoke a callback method on the original caller when processing is complete

The async call consumer code makes this concrete — on receiving a message, it checks the retry count, calls the target, and on exhaustion pushes to the sideline:

// asynccallconsumer/service.go (simplified)
if i < c.maxRunPerMessage {
    err = c.asyncCall(ctx, acall)
} else {
    // max retries exceeded — sideline the message
    err = c.sidelineProducer.Publish(ctx, c.acaller.SidelineCallerTopicId, orderKey, eReq)
}

Use Case 3: Scheduled API

The problem

Many platform features are time-bound: listing promotions that activate at a specific hour, subscription renewals at expiry, or notification campaigns scheduled in advance. Each product team solving this individually — with their own cron jobs, polling loops, or delay queues — creates operational sprawl.

How it works

The Scheduled API is a natural extension of the Async API. Instead of AsyncCall, the caller uses ScheduleAsyncCall with one or more scheduled_at timestamps. The Functional RPC stores the payload and schedule in PostgreSQL, sharded by an order key for distribution. The Scheduler service has a fleet of workers, each responsible for a shard range. Workers poll their shard at a configured interval, retrieve due payloads, and enqueue them onto the async-call Kafka topic for the Handler to process — giving the same retry and sideline guarantees as a regular async call.

The shard ID is derived from the order key (or a UUID if none is set) using a consistent hash, ensuring even distribution across scheduler workers while maintaining ordering guarantees for messages with the same key.

Keeping Messaging Reliable

Handling Failed Messages

Every subscription and async-call registration gets a dedicated sideline Kafka topic alongside the main one. Messages that exceed the retry limit land here rather than being dropped. Platform operators can inspect them — the full serialised protobuf envelope is stored — and replay them once the root cause is resolved. This is a first-class concern, not an afterthought.

Distributed Tracing

Every message carries a TrackingId — a serialised trace context extracted from the incoming gRPC metadata at publish time. When the Handler makes the downstream call, it deserialises this context and continues the trace span. The result is a single trace view that spans the synchronous user request and all subsequent async hops.

Monitoring and Alerts

With messages flowing between many services, we need to quickly know when something is slow or failing.

Get Edward Samuel Pasaribu’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The platform provides self-serve Grafana dashboards showing message publish and consumption rates, as well as Kafka consumer lag (messages waiting to be processed). These can be filtered by publisher, message type, and subscriber, making it easier for teams to identify where a problem is happening.

Simplified view of the Grafana dashboard for monitoring message publish and consumption rates.

Alerts are created automatically whenever a new subscriber or async caller is registered. Each team can adjust the alert threshold based on how much delay their use case can tolerate, and alerts are routed to the team that owns the service.

Self-Serve Onboarding

Registering a new publisher or subscriber requires a single Management API call. The Coordinator handles everything that follows: Kafka topic creation (with configured partition count, retention, and replication), Handler pod deployment on Kubernetes, and configuration refresh. Product teams have no Kafka or Kubernetes dependencies to manage.

By the Numbers

Since 2017, the Messaging Service has been the primary async communication backbone at Carousell. Here’s what it handles today:

• Handle ~5.5 Billion messages / month, at peak handles ~5000 messages/sec
• With latencies of p50: ~41ms end-to-end from publish to subscriber gRPC call and p99: ~458ms
• Covers 120+ microservices using the platform
• 3 communication patterns under a single API surface
• Hundreds of active subscriptions and async-call registrations across production
• Zero shared infra ownership between producer and consumer teams

Conclusion

Messaging started as a solution to a simple problem: allowing our growing number of services to communicate without depending too much on each other.

Almost a decade later, it handles billions of messages every month across more than 120 microservices. The technology underneath — Kafka, gRPC, PostgreSQL and Kubernetes — has been important, but the bigger lesson has been keeping the experience simple for the engineers using it.

For most teams, sending something asynchronously should feel almost as simple as making a normal service call. The complexity of delivery, retries, failures, scaling and monitoring belongs in the platform.

— Thanks to Praveen Sanap and Eric Feng Chao for their contribution to the Messaging Service.