Old Coordinator Leader keeps emitting stale metrics for a specific datasource

[aruraghuwanshi]

Affected Version

28.0.1

Description

After the coordinator leader election is concluded, sometimes the old coordinator leader keeps emitting stale metrics for a specific datasource, despite the new coordinator leader emitting the realtime metrics. This creates duplication and erroneous reporting.

Some details that were noted:

• Its always the high ingestion volume datasource that goes into this problem state with 2 coordinators emitting that metric (One stale and one actual). All other datasources' metrics are emitted only by the new coordinator-leader.
• No difference in logs emitted by the old coordinator leader for the problematic datasource vs the good ones.

Reference Image attached (Blue: Old coordinator metric for one specific datasource in a stuck/stale state; Green: New coordinator emitting the same metric with the realtime values)

[aruraghuwanshi]

Metrics reporter was marked closed for that datasource during the leader transition, but the stale metrics are still being emitted by the old-coordinator leader.

[kfaraz]

@aruraghuwanshi , I am not sure if you are referring to a metric emitted by Druid itself or some metric emitted by Kafka (since the logs you shared above indicate something originating in Kafka code).
If it's the former, can you please share the stale metric names that are coming from the old coordinator leader?

[aruraghuwanshi]

Attached two examples here. I'm referring to all the mentioned druid metrics in this Kafka section

[aruraghuwanshi]

Adding the metric names here for reference @kfaraz :
ingest/kafka/lag
ingest/kafka/maxLag
ingest/kafka/avgLag
ingest/kafka/partitionLag

[aruraghuwanshi]

For more context, we've faced this issue a few times before and the only resolution seems to be to kill the old-coordinator leader pod. Once that pod restarts and stabilizes the stale metric disappears and the real-values of the kafka lag metric(s) are only emitted by the current coordiantor leader.

[kfaraz]

Could you please check the service/heartbeat metric with dimension leader for the two coordinators and check if both of them are considering themselves to be leader at the same time?

Side note: Are you running coordinator and overlord as a single service?

[aruraghuwanshi]

Hey @kfaraz , unfortunately we're not emitting the heartbeat metric but we did confirm that we only had one active-leader at the time of this Incident, as per the logs shown here.

| Are you running coordinator and overlord as a single service?
Yes, that is accurate