Skip to content

[improve](spill) use batch_size session var to limit spill sort output block batch row count #50158

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 19, 2025

Conversation

jacktengg
Copy link
Contributor

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@jacktengg
Copy link
Contributor Author

run buildall

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@doris-robot
Copy link

TPC-H: Total hot run time: 33983 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4bd615e4a8d0cb736af393acebec593c4c689b76, data reload: false

------ Round 1 ----------------------------------
q1	26276	5020	5012	5012
q2	2074	277	185	185
q3	10427	1242	681	681
q4	10242	990	523	523
q5	7575	2400	2329	2329
q6	191	164	130	130
q7	930	734	628	628
q8	9314	1255	1096	1096
q9	6714	5139	5120	5120
q10	6854	2315	1863	1863
q11	492	277	266	266
q12	356	359	212	212
q13	17779	3639	3147	3147
q14	247	231	214	214
q15	527	483	484	483
q16	442	447	413	413
q17	591	856	354	354
q18	7550	7133	7276	7133
q19	1825	974	566	566
q20	318	328	225	225
q21	4011	3537	2442	2442
q22	1074	1030	961	961
Total cold run time: 115809 ms
Total hot run time: 33983 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5224	5068	5126	5068
q2	248	332	232	232
q3	2100	2641	2293	2293
q4	1411	1833	1423	1423
q5	4454	4371	4381	4371
q6	214	171	131	131
q7	1990	1895	1734	1734
q8	2620	2542	2518	2518
q9	7199	7187	7055	7055
q10	2958	3182	2747	2747
q11	587	508	495	495
q12	678	785	606	606
q13	3489	3890	3283	3283
q14	280	320	276	276
q15	520	487	478	478
q16	479	508	460	460
q17	1170	1617	1362	1362
q18	7838	7584	7495	7495
q19	793	796	943	796
q20	1980	2031	1866	1866
q21	5245	4856	4907	4856
q22	1087	1070	1028	1028
Total cold run time: 52564 ms
Total hot run time: 50573 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191355 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4bd615e4a8d0cb736af393acebec593c4c689b76, data reload: false

query1	1404	1106	1064	1064
query2	6261	1819	1828	1819
query3	11021	4493	4545	4493
query4	54457	24675	23162	23162
query5	5114	474	448	448
query6	343	222	207	207
query7	4900	478	277	277
query8	307	250	247	247
query9	5367	2526	2537	2526
query10	426	317	251	251
query11	15129	15008	14738	14738
query12	154	112	105	105
query13	1007	507	391	391
query14	10028	6357	6370	6357
query15	206	183	175	175
query16	7100	674	471	471
query17	1075	720	569	569
query18	1648	399	314	314
query19	179	179	168	168
query20	126	116	125	116
query21	208	158	112	112
query22	4337	4492	4337	4337
query23	34082	33140	33732	33140
query24	6611	2403	2381	2381
query25	452	488	410	410
query26	704	274	155	155
query27	2204	524	335	335
query28	2939	2143	2142	2142
query29	583	579	454	454
query30	268	228	199	199
query31	882	849	772	772
query32	81	65	70	65
query33	479	376	311	311
query34	774	875	537	537
query35	795	842	752	752
query36	959	1030	910	910
query37	115	114	79	79
query38	4225	4290	4215	4215
query39	1507	1426	1451	1426
query40	212	125	115	115
query41	98	57	52	52
query42	134	107	112	107
query43	499	501	491	491
query44	1341	823	811	811
query45	181	179	172	172
query46	835	1033	649	649
query47	1841	1886	1770	1770
query48	372	426	321	321
query49	685	504	429	429
query50	650	727	403	403
query51	4203	4346	4210	4210
query52	108	134	103	103
query53	229	262	192	192
query54	573	599	506	506
query55	85	81	80	80
query56	304	284	294	284
query57	1148	1168	1098	1098
query58	267	257	268	257
query59	2751	2758	2590	2590
query60	330	334	307	307
query61	132	132	142	132
query62	746	738	664	664
query63	237	186	183	183
query64	1760	1057	723	723
query65	4404	4254	4184	4184
query66	710	401	319	319
query67	15753	15577	15249	15249
query68	7610	879	500	500
query69	539	294	279	279
query70	1223	1117	1080	1080
query71	489	318	286	286
query72	5816	4756	4729	4729
query73	1493	614	359	359
query74	8991	9108	8731	8731
query75	3740	3196	2710	2710
query76	4236	1187	771	771
query77	623	348	272	272
query78	9880	10058	9181	9181
query79	5682	793	544	544
query80	735	502	428	428
query81	493	255	217	217
query82	657	124	93	93
query83	358	247	231	231
query84	292	100	77	77
query85	804	360	344	344
query86	376	293	281	281
query87	4383	4508	4330	4330
query88	2796	2252	2180	2180
query89	459	319	280	280
query90	1942	205	211	205
query91	135	147	108	108
query92	72	60	55	55
query93	2938	916	581	581
query94	685	412	298	298
query95	370	288	281	281
query96	475	548	276	276
query97	3125	3219	3120	3120
query98	227	210	207	207
query99	1468	1395	1290	1290
Total cold run time: 302157 ms
Total hot run time: 191355 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.44 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4bd615e4a8d0cb736af393acebec593c4c689b76, data reload: false

query1	0.04	0.04	0.02
query2	0.11	0.11	0.11
query3	0.26	0.19	0.19
query4	1.59	0.19	0.20
query5	0.61	0.60	0.60
query6	1.19	0.73	0.72
query7	0.02	0.02	0.02
query8	0.05	0.03	0.04
query9	0.58	0.52	0.51
query10	0.57	0.56	0.57
query11	0.16	0.11	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.60
query14	1.21	1.18	1.27
query15	0.88	0.87	0.85
query16	0.38	0.39	0.38
query17	1.01	1.07	1.02
query18	0.21	0.20	0.20
query19	1.89	1.76	1.77
query20	0.01	0.01	0.02
query21	15.42	0.89	0.57
query22	0.79	1.06	0.69
query23	15.00	1.38	0.58
query24	7.38	1.58	0.61
query25	0.44	0.21	0.08
query26	0.75	0.18	0.15
query27	0.05	0.05	0.05
query28	8.99	0.86	0.43
query29	12.54	4.03	3.36
query30	0.25	0.10	0.06
query31	2.83	0.58	0.39
query32	3.22	0.55	0.48
query33	3.00	3.08	3.06
query34	15.74	5.09	4.49
query35	4.53	4.52	4.50
query36	0.68	0.52	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.02
query40	0.17	0.13	0.14
query41	0.07	0.02	0.03
query42	0.03	0.02	0.02
query43	0.03	0.03	0.02
Total cold run time: 103.59 s
Total hot run time: 29.44 s

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 53.15% (14422/27132)
Line Coverage 42.02% (125004/297455)
Region Coverage 40.83% (63851/156379)
Branch Coverage 35.47% (32111/90534)

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Apr 18, 2025
Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Member

@mrhhsg mrhhsg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@yiguolei yiguolei merged commit 9fcbeca into apache:master Apr 19, 2025
25 of 27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants