Skip to content

[fix](bdbje) set maxMessageSize for bdbje #50213

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Apr 22, 2025

Conversation

dataroaring
Copy link
Contributor

default is cache size / 2. #50093 reduce memory consumed by fe via reducing cache size of bdbje.

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Apr 21, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring
Copy link
Contributor Author

run buildall

gavinchou
gavinchou previously approved these changes Apr 21, 2025
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 21, 2025
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@dataroaring
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Apr 21, 2025
@dataroaring
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 34685 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 11e7f7ceba864e144e4f26fed79258a7ba9d43a2, data reload: false

------ Round 1 ----------------------------------
q1	26657	5086	5115	5086
q2	2827	285	196	196
q3	11649	1267	692	692
q4	10231	992	536	536
q5	7536	2389	2351	2351
q6	181	163	133	133
q7	919	753	597	597
q8	9561	1299	1057	1057
q9	7384	5173	5201	5173
q10	6821	2358	1964	1964
q11	493	302	276	276
q12	370	379	239	239
q13	18052	3976	3467	3467
q14	259	255	237	237
q15	598	502	513	502
q16	480	472	421	421
q17	615	863	377	377
q18	9613	7320	7143	7143
q19	1663	965	539	539
q20	333	326	232	232
q21	4197	3400	2464	2464
q22	1073	1032	1003	1003
Total cold run time: 121512 ms
Total hot run time: 34685 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5235	5141	5106	5106
q2	252	327	237	237
q3	2229	2788	2504	2504
q4	1561	1918	1469	1469
q5	4562	4470	4455	4455
q6	222	172	132	132
q7	2049	1956	1795	1795
q8	2656	2539	2554	2539
q9	7190	7207	7105	7105
q10	2995	3184	2744	2744
q11	579	525	489	489
q12	691	790	644	644
q13	3519	3948	3334	3334
q14	291	307	278	278
q15	534	486	480	480
q16	476	503	464	464
q17	1151	1535	1393	1393
q18	7804	7552	7209	7209
q19	786	835	903	835
q20	2022	2010	1878	1878
q21	5397	4861	4711	4711
q22	1128	1081	1041	1041
Total cold run time: 53329 ms
Total hot run time: 50842 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192339 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 11e7f7ceba864e144e4f26fed79258a7ba9d43a2, data reload: false

query1	1397	1072	1091	1072
query2	6260	1793	1811	1793
query3	11145	4723	4670	4670
query4	25827	23906	23428	23428
query5	4090	619	462	462
query6	317	210	209	209
query7	3992	502	286	286
query8	295	242	251	242
query9	8491	2581	2568	2568
query10	477	338	273	273
query11	15197	15155	14797	14797
query12	162	109	111	109
query13	1583	531	399	399
query14	8539	6371	6160	6160
query15	210	192	177	177
query16	7227	661	470	470
query17	1232	704	559	559
query18	1982	431	349	349
query19	196	188	164	164
query20	129	130	125	125
query21	209	157	115	115
query22	4515	4611	4726	4611
query23	34643	33697	33463	33463
query24	8435	2426	2395	2395
query25	498	459	400	400
query26	1376	268	148	148
query27	2827	506	330	330
query28	4608	2144	2137	2137
query29	731	587	436	436
query30	281	213	186	186
query31	939	841	805	805
query32	70	59	62	59
query33	559	376	319	319
query34	916	886	512	512
query35	822	850	776	776
query36	977	993	914	914
query37	114	98	77	77
query38	4218	4369	4116	4116
query39	1463	1470	1442	1442
query40	217	121	108	108
query41	52	53	61	53
query42	120	106	103	103
query43	487	527	467	467
query44	1310	823	787	787
query45	177	178	175	175
query46	864	1056	646	646
query47	1814	1855	1785	1785
query48	391	414	310	310
query49	758	521	422	422
query50	696	701	405	405
query51	4239	4199	4184	4184
query52	106	109	97	97
query53	229	271	196	196
query54	584	587	515	515
query55	86	83	82	82
query56	292	289	307	289
query57	1176	1160	1096	1096
query58	271	277	271	271
query59	2660	2745	2635	2635
query60	326	336	308	308
query61	138	133	129	129
query62	793	736	648	648
query63	220	183	181	181
query64	4038	1027	694	694
query65	4303	4249	4209	4209
query66	1123	399	298	298
query67	15954	15610	15205	15205
query68	8254	880	513	513
query69	477	298	261	261
query70	1191	1178	1093	1093
query71	456	308	296	296
query72	5855	4665	4699	4665
query73	659	553	336	336
query74	8857	9071	8852	8852
query75	3934	3198	2716	2716
query76	3679	1204	763	763
query77	777	403	276	276
query78	9973	10048	9373	9373
query79	2562	816	560	560
query80	677	494	441	441
query81	484	249	220	220
query82	475	122	95	95
query83	292	250	233	233
query84	287	109	83	83
query85	803	359	319	319
query86	382	283	287	283
query87	4442	4467	4238	4238
query88	3652	2239	2197	2197
query89	402	321	283	283
query90	1838	209	214	209
query91	142	141	112	112
query92	79	65	56	56
query93	2323	953	573	573
query94	679	417	311	311
query95	381	298	283	283
query96	481	564	270	270
query97	3187	3236	3114	3114
query98	222	219	196	196
query99	1421	1388	1268	1268
Total cold run time: 279696 ms
Total hot run time: 192339 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.18 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 11e7f7ceba864e144e4f26fed79258a7ba9d43a2, data reload: false

query1	0.03	0.04	0.03
query2	0.12	0.11	0.11
query3	0.25	0.19	0.19
query4	1.59	0.19	0.20
query5	0.59	0.59	0.58
query6	1.20	0.72	0.71
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.58	0.52	0.50
query10	0.57	0.56	0.56
query11	0.16	0.11	0.10
query12	0.15	0.12	0.11
query13	0.62	0.60	0.60
query14	1.20	1.16	1.17
query15	0.89	0.86	0.83
query16	0.37	0.38	0.39
query17	1.03	1.01	1.02
query18	0.21	0.19	0.20
query19	1.86	1.83	1.81
query20	0.01	0.01	0.01
query21	15.40	0.90	0.55
query22	0.73	1.20	0.65
query23	15.00	1.38	0.62
query24	7.72	0.93	0.32
query25	0.28	0.12	0.12
query26	0.62	0.17	0.13
query27	0.05	0.05	0.05
query28	8.53	0.89	0.43
query29	12.62	4.00	3.32
query30	0.28	0.10	0.06
query31	2.83	0.59	0.39
query32	3.23	0.54	0.46
query33	3.07	3.11	3.15
query34	15.86	5.11	4.50
query35	4.56	4.58	4.56
query36	0.67	0.49	0.48
query37	0.09	0.06	0.07
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.18	0.15	0.12
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 103.44 s
Total hot run time: 29.18 s

Yongqiang YANG added 3 commits April 21, 2025 22:10
@dataroaring dataroaring force-pushed the bdbje-max-message-size branch from 11e7f7c to d53cca7 Compare April 21, 2025 14:10
@dataroaring
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33854 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d53cca7898fad10490bcfbfe7eb01accfe6661a7, data reload: false

------ Round 1 ----------------------------------
q1	26090	5119	4969	4969
q2	2065	284	187	187
q3	10408	1230	685	685
q4	10227	977	518	518
q5	7512	2385	2244	2244
q6	177	159	132	132
q7	894	749	608	608
q8	9305	1277	1219	1219
q9	6795	5013	5143	5013
q10	6865	2303	1901	1901
q11	465	273	275	273
q12	335	347	213	213
q13	17770	3620	3115	3115
q14	218	227	204	204
q15	539	494	488	488
q16	451	451	396	396
q17	585	848	359	359
q18	7483	7276	7179	7179
q19	1530	983	542	542
q20	319	330	220	220
q21	3882	3276	2448	2448
q22	1026	1017	941	941
Total cold run time: 114941 ms
Total hot run time: 33854 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5130	5093	5014	5014
q2	230	322	233	233
q3	2214	2756	2394	2394
q4	1452	1893	1420	1420
q5	4493	4458	4452	4452
q6	215	174	130	130
q7	2025	1941	1797	1797
q8	2638	2652	2527	2527
q9	7392	7247	7126	7126
q10	2974	3132	2740	2740
q11	572	515	486	486
q12	693	741	610	610
q13	3556	3850	3356	3356
q14	285	289	270	270
q15	512	487	471	471
q16	472	522	479	479
q17	1129	1538	1374	1374
q18	7723	7516	7380	7380
q19	759	787	834	787
q20	1980	1972	1850	1850
q21	5266	4832	4697	4697
q22	1054	1062	1051	1051
Total cold run time: 52764 ms
Total hot run time: 50644 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192304 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d53cca7898fad10490bcfbfe7eb01accfe6661a7, data reload: false

query1	1397	1069	1069	1069
query2	6192	1769	1785	1769
query3	11001	4520	4433	4433
query4	54910	25784	23786	23786
query5	5002	511	468	468
query6	354	203	194	194
query7	4936	488	279	279
query8	320	244	235	235
query9	6008	2576	2596	2576
query10	418	324	256	256
query11	15237	15112	14827	14827
query12	153	109	105	105
query13	1097	523	422	422
query14	10067	6345	6326	6326
query15	212	203	180	180
query16	7036	686	472	472
query17	1103	807	597	597
query18	1545	412	323	323
query19	197	196	171	171
query20	136	135	126	126
query21	217	126	111	111
query22	4459	4737	4430	4430
query23	34181	33189	33711	33189
query24	6614	2401	2351	2351
query25	482	460	396	396
query26	699	270	151	151
query27	2348	509	353	353
query28	3055	2181	2146	2146
query29	565	579	434	434
query30	277	217	193	193
query31	804	866	758	758
query32	68	66	63	63
query33	468	352	309	309
query34	764	868	533	533
query35	785	827	765	765
query36	931	993	917	917
query37	118	100	82	82
query38	4209	4296	4183	4183
query39	1489	1629	1423	1423
query40	211	133	105	105
query41	53	55	61	55
query42	122	107	106	106
query43	496	526	480	480
query44	1337	834	815	815
query45	179	170	162	162
query46	844	1032	649	649
query47	1840	1919	1822	1822
query48	400	419	301	301
query49	690	542	431	431
query50	666	695	413	413
query51	4162	4214	4180	4180
query52	113	113	102	102
query53	229	266	190	190
query54	587	582	519	519
query55	84	82	78	78
query56	294	294	300	294
query57	1185	1211	1120	1120
query58	258	261	261	261
query59	2676	2700	2601	2601
query60	319	331	330	330
query61	128	120	126	120
query62	799	737	662	662
query63	225	186	189	186
query64	1950	1045	683	683
query65	4392	4303	4254	4254
query66	709	437	300	300
query67	15891	15528	15306	15306
query68	8674	818	509	509
query69	533	301	260	260
query70	1190	1099	1102	1099
query71	506	327	291	291
query72	5832	4604	4464	4464
query73	1279	556	336	336
query74	8957	8985	8983	8983
query75	3890	3177	2648	2648
query76	4349	1198	750	750
query77	602	363	277	277
query78	10085	10104	9326	9326
query79	2290	806	567	567
query80	697	500	454	454
query81	495	274	211	211
query82	427	125	93	93
query83	276	237	230	230
query84	286	126	87	87
query85	794	341	309	309
query86	403	307	273	273
query87	4421	4382	4272	4272
query88	3358	2222	2271	2222
query89	400	320	279	279
query90	1819	201	199	199
query91	137	137	108	108
query92	73	57	65	57
query93	1869	917	586	586
query94	662	413	306	306
query95	373	283	288	283
query96	480	554	273	273
query97	3181	3209	3148	3148
query98	222	213	222	213
query99	1440	1436	1265	1265
Total cold run time: 300693 ms
Total hot run time: 192304 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.08 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d53cca7898fad10490bcfbfe7eb01accfe6661a7, data reload: false

query1	0.03	0.04	0.03
query2	0.12	0.11	0.12
query3	0.25	0.20	0.19
query4	1.60	0.20	0.19
query5	0.60	0.59	0.60
query6	1.21	0.71	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.03
query9	0.57	0.53	0.51
query10	0.56	0.58	0.56
query11	0.14	0.11	0.11
query12	0.15	0.11	0.11
query13	0.60	0.59	0.60
query14	1.18	1.18	1.19
query15	0.88	0.85	0.85
query16	0.38	0.38	0.37
query17	0.98	1.02	1.05
query18	0.21	0.19	0.19
query19	1.81	1.75	1.78
query20	0.01	0.01	0.00
query21	15.41	0.91	0.56
query22	0.76	1.30	0.94
query23	14.68	1.39	0.66
query24	6.73	1.05	1.63
query25	0.49	0.21	0.12
query26	0.53	0.16	0.14
query27	0.06	0.05	0.06
query28	9.78	0.85	0.44
query29	12.67	3.93	3.32
query30	0.26	0.08	0.06
query31	2.84	0.57	0.40
query32	3.23	0.54	0.48
query33	2.98	3.04	3.00
query34	15.76	5.03	4.44
query35	4.51	4.53	4.52
query36	0.66	0.50	0.49
query37	0.09	0.07	0.06
query38	0.05	0.03	0.03
query39	0.02	0.02	0.02
query40	0.16	0.14	0.13
query41	0.09	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 103.18 s
Total hot run time: 30.08 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 22, 2025
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@dataroaring dataroaring merged commit 163834a into apache:master Apr 22, 2025
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants