[feat](cloud) Add a config for cloud txn lazy commit fuzzy test #50314

SWJTU-ZhangLei · 2025-04-23T02:07:08Z

when setting enable_cloud_txn_lazy_commit_fuzzy_test=true, commit_txn will be fifty percent using commit_txn_eventually

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-04-23T02:07:15Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

SWJTU-ZhangLei · 2025-04-23T02:07:19Z

run buildall

gavinchou · 2025-04-23T02:14:16Z

cloud/src/meta-service/meta_service_txn.cpp

+    if (bd(gen)) {
+        return true;
+    }
+    return false;


Suggested change

if (bd(gen)) {

return true;

}

return false;

return bd(gen);

doris-robot · 2025-04-23T02:16:09Z

TeamCity cloud ut coverage result:
Function Coverage: 82.95% (1095/1320)
Line Coverage: 65.69% (18281/27830)
Region Coverage: 65.08% (9057/13916)
Branch Coverage: 55.06% (4877/8858)
Coverage Report: http://coverage.selectdb-in.cc/coverage/40d8bc814a75aa2550285fbe85fa8bf8a8e9aed8_40d8bc814a75aa2550285fbe85fa8bf8a8e9aed8_cloud/report/index.html

SWJTU-ZhangLei · 2025-04-23T02:28:26Z

run buildall

doris-robot · 2025-04-23T02:34:24Z

TeamCity cloud ut coverage result:
Function Coverage: 82.95% (1095/1320)
Line Coverage: 65.65% (18269/27827)
Region Coverage: 65.05% (9050/13913)
Branch Coverage: 55.06% (4876/8856)
Coverage Report: http://coverage.selectdb-in.cc/coverage/50be89500fd11be5a468d1dde0a5d10ede9949f3_50be89500fd11be5a468d1dde0a5d10ede9949f3_cloud/report/index.html

SWJTU-ZhangLei · 2025-04-23T12:32:14Z

run buildall

doris-robot · 2025-04-23T12:45:24Z

TeamCity cloud ut coverage result:
Function Coverage: 83.08% (1100/1324)
Line Coverage: 65.97% (18411/27909)
Region Coverage: 65.44% (9130/13951)
Branch Coverage: 55.33% (4907/8868)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3d7eed96191b985b27c097ccde4bd2d6b92168ff_3d7eed96191b985b27c097ccde4bd2d6b92168ff_cloud/report/index.html

SWJTU-ZhangLei · 2025-04-24T14:19:00Z

run buildall

doris-robot · 2025-04-24T14:28:07Z

TeamCity cloud ut coverage result:
Function Coverage: 83.08% (1100/1324)
Line Coverage: 65.84% (18379/27913)
Region Coverage: 65.28% (9111/13957)
Branch Coverage: 55.23% (4900/8872)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a3f9a8eeb20ae1ba7035ac583f31ad6c8f75024b_a3f9a8eeb20ae1ba7035ac583f31ad6c8f75024b_cloud/report/index.html

gavinchou · 2025-04-25T06:48:08Z

cloud/src/meta-service/meta_service_txn.cpp

            (request->has_is_2pc() && !request->is_2pc() && request->has_enable_txn_lazy_commit() &&
             request->enable_txn_lazy_commit() && config::enable_cloud_txn_lazy_commit);

-    if (!allow_txn_lazy_commit ||
-        (tmp_rowsets_meta.size() <= config::txn_lazy_commit_rowsets_thresold)) {
+    if (force_txn_lazy_commit() &&


while (!allow_txn_lazy_commit && (!enable_txn_lazy_commit_feature || (tmp_rowsets_meta.size() <= config::txn_lazy_commit_rowsets_thresold))) { if (force_lazy_commit()) { LOG(); break; } commit_txn_immediately(request, response, txn_kv_, txn_lazy_committer_, code, msg, instance_id, db_id, tmp_rowsets_meta, err); if ((MetaServiceCode::OK == code) || (TxnErrorCode::TXN_BYTES_TOO_LARGE != err) || !enable_txn_lazy_commit_feature) { if (err == TxnErrorCode::TXN_BYTES_TOO_LARGE) { msg += ", likely due to committing too many tablets. " "Please reduce the number of partitions involved in the load."; } return; } DCHECK(code != MetaServiceCode::OK); DCHECK(enable_txn_lazy_commit_feature); DCHECK(err == TxnErrorCode::TXN_BYTES_TOO_LARGE); LOG(INFO) << "txn_id=" << txn_id << " fallthrough commit_txn_eventually"; break; }

gavinchou · 2025-04-25T06:50:38Z

cloud/src/meta-service/meta_service_txn.cpp

@@ -2514,6 +2514,20 @@ void commit_txn_with_sub_txn(const CommitTxnRequest* request, CommitTxnResponse*
    response->mutable_txn_info()->CopyFrom(txn_info);
 } // end commit_txn_with_sub_txn

+static bool fuzzy_random() {
+    std::mt19937 gen {std::random_device {}()};


return (steady_clock():now().count() & 0x01)

return (steady_clock():now().count() & 0x01)

done

SWJTU-ZhangLei · 2025-04-25T07:43:19Z

run buildall

doris-robot · 2025-04-25T07:56:23Z

TeamCity cloud ut coverage result:
Function Coverage: 83.08% (1100/1324)
Line Coverage: 65.88% (18396/27923)
Region Coverage: 65.32% (9120/13962)
Branch Coverage: 55.27% (4905/8874)
Coverage Report: http://coverage.selectdb-in.cc/coverage/80c1c608c90c3f2caf9114d0f4fc01c50ae471ec_80c1c608c90c3f2caf9114d0f4fc01c50ae471ec_cloud/report/index.html

* when setting `enable_cloud_txn_lazy_commit_fuzzy_test=true`, `commit_txn` will be fifty percent using `commit_txn_eventually`

SWJTU-ZhangLei · 2025-04-25T09:39:15Z

run buildall

doris-robot · 2025-04-25T09:50:14Z

TeamCity cloud ut coverage result:
Function Coverage: 83.08% (1100/1324)
Line Coverage: 65.86% (18389/27923)
Region Coverage: 65.28% (9114/13962)
Branch Coverage: 55.22% (4900/8874)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b04ee720e3eb0ec8a25329ae79de3c0df25c65f3_b04ee720e3eb0ec8a25329ae79de3c0df25c65f3_cloud/report/index.html

SWJTU-ZhangLei · 2025-04-25T23:59:39Z

run cloud_p0

dataroaring

LGTM

github-actions · 2025-04-27T07:59:40Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-04-27T07:59:42Z

PR approved by anyone and no changes requested.

gavinchou · 2025-04-27T08:10:59Z

cloud/src/meta-service/meta_service_txn.cpp

+    while ((!enable_txn_lazy_commit_feature ||
+            (tmp_rowsets_meta.size() <= config::txn_lazy_commit_rowsets_thresold))) {
+        if (force_txn_lazy_commit()) {
+            LOG(INFO) << "fuzzy test force_txn_lazy_commit";


print txnId and tablet id etc

print txnId and tablet id etc

done

gavinchou · 2025-04-27T08:12:42Z

cloud/test/txn_lazy_commit_test.cpp

+        }
+    }
+    LOG(INFO) << "force_txn_lazy_commit counter: " << counter;
+    ASSERT_GT(counter, 0);


it should be [3000 - 7000]

it should be [3000 - 7000]

done

SWJTU-ZhangLei · 2025-04-27T08:55:49Z

run buildall

doris-robot · 2025-04-27T09:13:14Z

TeamCity cloud ut coverage result:
Function Coverage: 83.08% (1100/1324)
Line Coverage: 65.86% (18393/27928)
Region Coverage: 65.30% (9119/13964)
Branch Coverage: 55.29% (4906/8874)
Coverage Report: http://coverage.selectdb-in.cc/coverage/98f7f9c13b2619a2b943a713bdeec50c719258e1_98f7f9c13b2619a2b943a713bdeec50c719258e1_cloud/report/index.html

SWJTU-ZhangLei · 2025-04-27T13:51:24Z

run cloud_p0

SWJTU-ZhangLei · 2025-04-28T01:53:27Z

run cloud_p0

github-actions · 2025-04-28T03:21:32Z

PR approved by at least one committer and no changes requested.

SWJTU-ZhangLei · 2025-04-28T03:55:57Z

run cloud_p0

SWJTU-ZhangLei · 2025-04-28T06:04:01Z

run cloud_p0

dataroaring

LGTM

…he#50314) * when setting `enable_cloud_txn_lazy_commit_fuzzy_test=true`, `commit_txn` will be fifty percent using `commit_txn_eventually`

…) (#50544) * when setting `enable_cloud_txn_lazy_commit_fuzzy_test=true`, `commit_txn` will be fifty percent using `commit_txn_eventually`

…he#50314) * when setting `enable_cloud_txn_lazy_commit_fuzzy_test=true`, `commit_txn` will be fifty percent using `commit_txn_eventually`

SWJTU-ZhangLei requested review from gavinchou, dataroaring and w41ter as code owners April 23, 2025 02:07

gavinchou reviewed Apr 23, 2025

View reviewed changes

SWJTU-ZhangLei force-pushed the master-fix11 branch from cabaffb to 50be895 Compare April 23, 2025 02:28

dataroaring added the dev/3.0.x label Apr 23, 2025

gavinchou reviewed Apr 25, 2025

View reviewed changes

SWJTU-ZhangLei force-pushed the master-fix11 branch from a3f9a8e to 80c1c60 Compare April 25, 2025 07:42

SWJTU-ZhangLei added 4 commits April 25, 2025 17:38

[feat](cloud) Add a config for cloud txn lazy commit fuzzy test

d9c6883

* when setting `enable_cloud_txn_lazy_commit_fuzzy_test=true`, `commit_txn` will be fifty percent using `commit_txn_eventually`

Fix some comments

c56ede1

Fix some comments

ea6679d

fix flink_connector_response case error msg not match

b04ee72

SWJTU-ZhangLei force-pushed the master-fix11 branch from 4964f2e to b04ee72 Compare April 25, 2025 09:39

dataroaring previously approved these changes Apr 27, 2025

View reviewed changes

dataroaring added the p0_b label Apr 27, 2025

github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 27, 2025

github-actions bot added the reviewed label Apr 27, 2025

gavinchou reviewed Apr 27, 2025

View reviewed changes

Fix some comments

98f7f9c

SWJTU-ZhangLei dismissed dataroaring’s stale review via 98f7f9c April 27, 2025 08:55

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Apr 27, 2025

gavinchou approved these changes Apr 28, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 28, 2025

dataroaring approved these changes Apr 28, 2025

View reviewed changes

dataroaring merged commit bff7c6a into apache:master Apr 28, 2025
28 of 31 checks passed

github-actions bot added the dev/3.0.x-conflict label Apr 28, 2025

dataroaring added dev/3.0.6-merged and removed dev/3.0.x dev/3.0.x-conflict labels Apr 30, 2025

dataroaring mentioned this pull request Apr 30, 2025

[feat](cloud) Add a config for cloud txn lazy commit fuzzy test (#50314) #50544

Merged

16 tasks

[feat](cloud) Add a config for cloud txn lazy commit fuzzy test #50314

[feat](cloud) Add a config for cloud txn lazy commit fuzzy test #50314

Uh oh!

Conversation

SWJTU-ZhangLei commented Apr 23, 2025

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Apr 23, 2025

Uh oh!

SWJTU-ZhangLei commented Apr 23, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

doris-robot commented Apr 23, 2025

Uh oh!

SWJTU-ZhangLei commented Apr 23, 2025

Uh oh!

doris-robot commented Apr 23, 2025

Uh oh!

SWJTU-ZhangLei commented Apr 23, 2025

Uh oh!

doris-robot commented Apr 23, 2025

Uh oh!

SWJTU-ZhangLei commented Apr 24, 2025

Uh oh!

doris-robot commented Apr 24, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SWJTU-ZhangLei commented Apr 25, 2025

Uh oh!

doris-robot commented Apr 25, 2025

Uh oh!

SWJTU-ZhangLei commented Apr 25, 2025

Uh oh!

doris-robot commented Apr 25, 2025

Uh oh!

SWJTU-ZhangLei commented Apr 25, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 27, 2025

Uh oh!

github-actions bot commented Apr 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SWJTU-ZhangLei commented Apr 27, 2025

Uh oh!

doris-robot commented Apr 27, 2025

Uh oh!

SWJTU-ZhangLei commented Apr 27, 2025

Uh oh!

SWJTU-ZhangLei commented Apr 28, 2025

Uh oh!

github-actions bot commented Apr 28, 2025

Uh oh!

SWJTU-ZhangLei commented Apr 28, 2025

Uh oh!

SWJTU-ZhangLei commented Apr 28, 2025

Uh oh!

dataroaring left a comment