Skip to content

[feat](cloud) Add a config for cloud txn lazy commit fuzzy test #50314

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Apr 28, 2025

Conversation

SWJTU-ZhangLei
Copy link
Contributor

  • when setting enable_cloud_txn_lazy_commit_fuzzy_test=true, commit_txn will be fifty percent using commit_txn_eventually

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@SWJTU-ZhangLei
Copy link
Contributor Author

run buildall

Comment on lines 2521 to 2524
if (bd(gen)) {
return true;
}
return false;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (bd(gen)) {
return true;
}
return false;
return bd(gen);

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@doris-robot
Copy link

TeamCity cloud ut coverage result:
Function Coverage: 82.95% (1095/1320)
Line Coverage: 65.69% (18281/27830)
Region Coverage: 65.08% (9057/13916)
Branch Coverage: 55.06% (4877/8858)
Coverage Report: http://coverage.selectdb-in.cc/coverage/40d8bc814a75aa2550285fbe85fa8bf8a8e9aed8_40d8bc814a75aa2550285fbe85fa8bf8a8e9aed8_cloud/report/index.html

@SWJTU-ZhangLei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity cloud ut coverage result:
Function Coverage: 82.95% (1095/1320)
Line Coverage: 65.65% (18269/27827)
Region Coverage: 65.05% (9050/13913)
Branch Coverage: 55.06% (4876/8856)
Coverage Report: http://coverage.selectdb-in.cc/coverage/50be89500fd11be5a468d1dde0a5d10ede9949f3_50be89500fd11be5a468d1dde0a5d10ede9949f3_cloud/report/index.html

@SWJTU-ZhangLei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity cloud ut coverage result:
Function Coverage: 83.08% (1100/1324)
Line Coverage: 65.97% (18411/27909)
Region Coverage: 65.44% (9130/13951)
Branch Coverage: 55.33% (4907/8868)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3d7eed96191b985b27c097ccde4bd2d6b92168ff_3d7eed96191b985b27c097ccde4bd2d6b92168ff_cloud/report/index.html

@SWJTU-ZhangLei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity cloud ut coverage result:
Function Coverage: 83.08% (1100/1324)
Line Coverage: 65.84% (18379/27913)
Region Coverage: 65.28% (9111/13957)
Branch Coverage: 55.23% (4900/8872)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a3f9a8eeb20ae1ba7035ac583f31ad6c8f75024b_a3f9a8eeb20ae1ba7035ac583f31ad6c8f75024b_cloud/report/index.html

(request->has_is_2pc() && !request->is_2pc() && request->has_enable_txn_lazy_commit() &&
request->enable_txn_lazy_commit() && config::enable_cloud_txn_lazy_commit);

if (!allow_txn_lazy_commit ||
(tmp_rowsets_meta.size() <= config::txn_lazy_commit_rowsets_thresold)) {
if (force_txn_lazy_commit() &&
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

while (!allow_txn_lazy_commit &&
    (!enable_txn_lazy_commit_feature ||
     (tmp_rowsets_meta.size() <= config::txn_lazy_commit_rowsets_thresold))) {

    if (force_lazy_commit()) {
      LOG();
      break;
    }

    commit_txn_immediately(request, response, txn_kv_, txn_lazy_committer_, code, msg,
                           instance_id, db_id, tmp_rowsets_meta, err);

    if ((MetaServiceCode::OK == code) || (TxnErrorCode::TXN_BYTES_TOO_LARGE != err) ||
        !enable_txn_lazy_commit_feature) {
        if (err == TxnErrorCode::TXN_BYTES_TOO_LARGE) {
            msg += ", likely due to committing too many tablets. "
                   "Please reduce the number of partitions involved in the load.";
        }
        return;
    }
    DCHECK(code != MetaServiceCode::OK);
    DCHECK(enable_txn_lazy_commit_feature);
    DCHECK(err == TxnErrorCode::TXN_BYTES_TOO_LARGE);
    LOG(INFO) << "txn_id=" << txn_id << " fallthrough commit_txn_eventually";
    break;
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@@ -2514,6 +2514,20 @@ void commit_txn_with_sub_txn(const CommitTxnRequest* request, CommitTxnResponse*
response->mutable_txn_info()->CopyFrom(txn_info);
} // end commit_txn_with_sub_txn

static bool fuzzy_random() {
std::mt19937 gen {std::random_device {}()};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

return (steady_clock():now().count() & 0x01)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

return (steady_clock():now().count() & 0x01)

done

@SWJTU-ZhangLei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity cloud ut coverage result:
Function Coverage: 83.08% (1100/1324)
Line Coverage: 65.88% (18396/27923)
Region Coverage: 65.32% (9120/13962)
Branch Coverage: 55.27% (4905/8874)
Coverage Report: http://coverage.selectdb-in.cc/coverage/80c1c608c90c3f2caf9114d0f4fc01c50ae471ec_80c1c608c90c3f2caf9114d0f4fc01c50ae471ec_cloud/report/index.html

* when setting `enable_cloud_txn_lazy_commit_fuzzy_test=true`, `commit_txn` will be
  fifty percent using `commit_txn_eventually`
@SWJTU-ZhangLei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity cloud ut coverage result:
Function Coverage: 83.08% (1100/1324)
Line Coverage: 65.86% (18389/27923)
Region Coverage: 65.28% (9114/13962)
Branch Coverage: 55.22% (4900/8874)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b04ee720e3eb0ec8a25329ae79de3c0df25c65f3_b04ee720e3eb0ec8a25329ae79de3c0df25c65f3_cloud/report/index.html

@SWJTU-ZhangLei
Copy link
Contributor Author

run cloud_p0

dataroaring
dataroaring previously approved these changes Apr 27, 2025
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 27, 2025
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

while ((!enable_txn_lazy_commit_feature ||
(tmp_rowsets_meta.size() <= config::txn_lazy_commit_rowsets_thresold))) {
if (force_txn_lazy_commit()) {
LOG(INFO) << "fuzzy test force_txn_lazy_commit";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

print txnId and tablet id etc

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

print txnId and tablet id etc

done

}
}
LOG(INFO) << "force_txn_lazy_commit counter: " << counter;
ASSERT_GT(counter, 0);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it should be [3000 - 7000]

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it should be [3000 - 7000]

done

@SWJTU-ZhangLei
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Apr 27, 2025
@doris-robot
Copy link

TeamCity cloud ut coverage result:
Function Coverage: 83.08% (1100/1324)
Line Coverage: 65.86% (18393/27928)
Region Coverage: 65.30% (9119/13964)
Branch Coverage: 55.29% (4906/8874)
Coverage Report: http://coverage.selectdb-in.cc/coverage/98f7f9c13b2619a2b943a713bdeec50c719258e1_98f7f9c13b2619a2b943a713bdeec50c719258e1_cloud/report/index.html

@SWJTU-ZhangLei
Copy link
Contributor Author

run cloud_p0

1 similar comment
@SWJTU-ZhangLei
Copy link
Contributor Author

run cloud_p0

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 28, 2025
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@SWJTU-ZhangLei
Copy link
Contributor Author

run cloud_p0

1 similar comment
@SWJTU-ZhangLei
Copy link
Contributor Author

run cloud_p0

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit bff7c6a into apache:master Apr 28, 2025
28 of 31 checks passed
SWJTU-ZhangLei added a commit to SWJTU-ZhangLei/incubator-doris that referenced this pull request Apr 29, 2025
…he#50314)

* when setting `enable_cloud_txn_lazy_commit_fuzzy_test=true`,
`commit_txn` will be fifty percent using `commit_txn_eventually`
dataroaring pushed a commit that referenced this pull request Apr 30, 2025
…) (#50544)

* when setting `enable_cloud_txn_lazy_commit_fuzzy_test=true`,
`commit_txn` will be fifty percent using `commit_txn_eventually`
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
…he#50314)

* when setting `enable_cloud_txn_lazy_commit_fuzzy_test=true`,
`commit_txn` will be fifty percent using `commit_txn_eventually`
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.6-merged p0_b reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants