Skip to content

Druid Coordinator stop assign segments after the zk restart #17807

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
GaryChow-LGTM opened this issue Mar 17, 2025 · 1 comment
Open

Druid Coordinator stop assign segments after the zk restart #17807

GaryChow-LGTM opened this issue Mar 17, 2025 · 1 comment

Comments

@GaryChow-LGTM
Copy link

GaryChow-LGTM commented Mar 17, 2025

Affected Version

Druid 25.0
Zookeeper version: 3.5.10

Description

  1. Zookeeper restart due to the node crash in 05:50 UTC
  2. Coordinator log
    2025-03-17T05:50:23,080 INFO [LeaderSelector[/druid/coordinator/_COORDINATOR]] org.apache.druid.server.coordinator.DruidCoordinator - I am no longer the leader... 2025-03-17T05:50:24,370 INFO [LeaderSelector[/druid/coordinator/_COORDINATOR]] org.apache.druid.server.coordinator.DruidCoordinator - I am the leader of the coordinators, all must bow! Starting coordination in [PT30S].
  3. There are no new segment assignments, but the following log is present with no error logs:
    Polled and found 201 rule(s) for 193 datasource(s).

I have attempted the following approaches to recover from the issue:

  1. Restarted the coordinator leader and follower, but this did not help.
  2. Restarted the Zookeeper follower, but this did not help.
  3. Restarted the Zookeeper leader, which resolved the issue.
@gianm
Copy link
Contributor

gianm commented Apr 1, 2025

It seems like some kind of issue with leader election. It's unfortunately hard to debug from just this provided information. It would possibly be helpful to see more logs. If this ever happens again, it's also possibly helpful to see a thread dump of the apparently-stuck Coordinator before restarting it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants