Skip to content

[cherry-pick](branch-3.0) Pick "[Enhancement](compaction) Try get global lock when execute compaction (#49882)" #50432

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: branch-3.0
Choose a base branch
from

Conversation

Yukang-Lian
Copy link
Collaborator

Pick #49882

Background:
In cloud mode, compaction tasks for the same tablet may be scheduled across multiple BEs. To ensure that only one BE can execute a compaction task for a given tablet at a time, a global locking mechanism is used.

During compaction preparation, tablet and compaction information is written as key-value pairs to the metadata service. A background thread periodically renews the lease. Other BEs can only perform compaction on a tablet when the KV entry has expired or doesn't exist, ensuring that a tablet's compaction occurs on only one BE at a time.

Problem:
Compaction tasks are processed through a thread pool. Currently, we first prepare compaction and acquire the global lock before queueing the task. If a BE is under heavy compaction pressure with all threads occupied, tablets may wait in the queue for extended periods. Meanwhile, other idle BEs cannot perform compaction on these tablets because they cannot acquire the global lock, leading to resource imbalance with some BEs starved and others overloaded.

Solution:
To address this issue, we'll modify the workflow to queue tasks first, then attempt to acquire the lock only when the task is about to be executed. This ensures that even if a tablet's compaction task is queued on one BE, another idle BE can still perform compaction on that tablet, resulting in better resource utilization across the cluster.

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

…apache#49882)

Background:
In cloud mode, compaction tasks for the same tablet may be scheduled
across multiple BEs. To ensure that only one BE can execute a compaction
task for a given tablet at a time, a global locking mechanism is used.

During compaction preparation, tablet and compaction information is
written as key-value pairs to the metadata service. A background thread
periodically renews the lease. Other BEs can only perform compaction on
a tablet when the KV entry has expired or doesn't exist, ensuring that a
tablet's compaction occurs on only one BE at a time.

Problem:
Compaction tasks are processed through a thread pool. Currently, we
first prepare compaction and acquire the global lock before queueing the
task. If a BE is under heavy compaction pressure with all threads
occupied, tablets may wait in the queue for extended periods. Meanwhile,
other idle BEs cannot perform compaction on these tablets because they
cannot acquire the global lock, leading to resource imbalance with some
BEs starved and others overloaded.

Solution:
To address this issue, we'll modify the workflow to queue tasks first,
then attempt to acquire the lock only when the task is about to be
executed. This ensures that even if a tablet's compaction task is queued
on one BE, another idle BE can still perform compaction on that tablet,
resulting in better resource utilization across the cluster.
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Yukang-Lian
Copy link
Collaborator Author

run buildall

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants