Skip to content

[opt](paimon/iceberg)upgrade paimon to 1.0.1, iceberg to 1.6.1 #49280

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Apr 25, 2025
Merged

[opt](paimon/iceberg)upgrade paimon to 1.0.1, iceberg to 1.6.1 #49280

merged 3 commits into from
Apr 25, 2025

Conversation

wuwenchi
Copy link
Contributor

@wuwenchi wuwenchi commented Mar 20, 2025 •
edited
Loading

@wuwenchi wuwenchi requested a review from dataroaring as a code owner March 20, 2025 01:56
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@wuwenchi
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40086 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 028b5ec95e7b08a2287c72a0953a342a7594fa08, data reload: false

------ Round 1 ----------------------------------
q1	17575	6878	6642	6642
q2	2064	173	178	173
q3	10640	1077	1129	1077
q4	10536	731	676	676
q5	7732	2845	2785	2785
q6	220	135	134	134
q7	1011	611	605	605
q8	9356	1932	2020	1932
q9	6691	6388	6403	6388
q10	7067	2211	2290	2211
q11	470	267	279	267
q12	412	215	210	210
q13	17800	3021	3067	3021
q14	246	204	206	204
q15	521	473	467	467
q16	686	583	591	583
q17	992	580	584	580
q18	7355	6685	6723	6685
q19	1500	1030	989	989
q20	472	209	196	196
q21	4223	3276	3390	3276
q22	1077	995	985	985
Total cold run time: 108646 ms
Total hot run time: 40086 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6669	6594	6650	6594
q2	325	235	230	230
q3	2901	2812	3050	2812
q4	2040	1830	1792	1792
q5	5822	5828	5797	5797
q6	216	129	134	129
q7	2248	1852	1809	1809
q8	3429	3627	3581	3581
q9	8877	8993	8892	8892
q10	3602	3538	3535	3535
q11	626	505	486	486
q12	829	618	614	614
q13	11024	3371	3250	3250
q14	316	282	292	282
q15	511	459	470	459
q16	674	650	658	650
q17	1854	1668	1612	1612
q18	8457	7882	8237	7882
q19	1714	1543	1539	1539
q20	2163	1994	1937	1937
q21	5804	5681	5675	5675
q22	1157	1083	1045	1045
Total cold run time: 71258 ms
Total hot run time: 60602 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196500 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 028b5ec95e7b08a2287c72a0953a342a7594fa08, data reload: false

query1	1258	911	889	889
query2	6259	2115	2080	2080
query3	10860	4385	4195	4195
query4	62767	28395	23337	23337
query5	5237	456	455	455
query6	420	182	170	170
query7	5480	320	309	309
query8	299	219	211	211
query9	8476	2591	2588	2588
query10	458	265	247	247
query11	17526	15091	15522	15091
query12	145	104	103	103
query13	1435	473	454	454
query14	10652	7101	7294	7101
query15	205	181	176	176
query16	7142	480	462	462
query17	1132	604	609	604
query18	1879	341	328	328
query19	228	169	166	166
query20	117	108	107	107
query21	206	103	103	103
query22	4816	4566	5001	4566
query23	34284	34074	33654	33654
query24	6186	2925	3008	2925
query25	537	438	464	438
query26	658	170	169	169
query27	1766	355	352	352
query28	4503	2452	2436	2436
query29	671	442	451	442
query30	247	166	168	166
query31	945	821	825	821
query32	70	55	59	55
query33	431	304	296	296
query34	915	501	514	501
query35	867	734	741	734
query36	1087	973	973	973
query37	121	69	66	66
query38	4263	3977	3992	3977
query39	1523	1486	1465	1465
query40	212	100	102	100
query41	49	51	49	49
query42	114	99	100	99
query43	546	492	500	492
query44	1205	819	821	819
query45	184	174	172	172
query46	1161	725	730	725
query47	2042	1885	1946	1885
query48	473	399	394	394
query49	708	402	384	384
query50	831	428	441	428
query51	7336	7108	7149	7108
query52	101	88	91	88
query53	264	187	183	183
query54	580	454	466	454
query55	79	78	82	78
query56	268	256	247	247
query57	1305	1152	1176	1152
query58	215	208	214	208
query59	3297	2945	2858	2858
query60	279	260	248	248
query61	107	129	106	106
query62	762	659	669	659
query63	217	188	184	184
query64	1391	664	635	635
query65	3287	3192	3199	3192
query66	695	295	294	294
query67	15991	15707	15587	15587
query68	4388	585	570	570
query69	436	262	256	256
query70	1207	1158	1033	1033
query71	345	266	260	260
query72	6344	4103	3982	3982
query73	757	352	359	352
query74	10349	9280	8988	8988
query75	3349	2650	2663	2650
query76	1938	1095	1065	1065
query77	490	266	262	262
query78	10557	9677	9642	9642
query79	1098	603	595	595
query80	784	430	436	430
query81	511	243	233	233
query82	465	87	88	87
query83	166	144	144	144
query84	282	86	80	80
query85	833	322	296	296
query86	335	300	305	300
query87	4427	4391	4195	4195
query88	3957	2414	2392	2392
query89	417	288	288	288
query90	2070	181	187	181
query91	184	149	152	149
query92	62	49	51	49
query93	1320	542	539	539
query94	747	289	282	282
query95	353	248	256	248
query96	607	290	281	281
query97	3312	3176	3166	3166
query98	210	205	198	198
query99	1526	1267	1270	1267
Total cold run time: 313596 ms
Total hot run time: 196500 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.71 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 028b5ec95e7b08a2287c72a0953a342a7594fa08, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.04	0.03
query3	0.23	0.07	0.06
query4	1.63	0.10	0.10
query5	0.54	0.53	0.51
query6	1.13	0.73	0.74
query7	0.02	0.02	0.01
query8	0.05	0.04	0.03
query9	0.55	0.50	0.49
query10	0.55	0.55	0.56
query11	0.13	0.11	0.12
query12	0.13	0.11	0.11
query13	0.61	0.60	0.60
query14	2.72	2.72	2.75
query15	0.88	0.82	0.82
query16	0.38	0.40	0.38
query17	1.07	1.07	1.05
query18	0.22	0.22	0.21
query19	1.87	1.76	1.97
query20	0.01	0.01	0.01
query21	15.36	0.59	0.58
query22	2.21	1.83	1.78
query23	16.88	1.16	0.76
query24	3.33	0.73	0.92
query25	0.27	0.17	0.13
query26	0.34	0.14	0.14
query27	0.04	0.05	0.03
query28	10.61	0.52	0.44
query29	12.56	3.22	3.24
query30	0.24	0.06	0.06
query31	2.87	0.38	0.37
query32	3.26	0.45	0.45
query33	2.92	2.99	3.00
query34	16.99	4.46	4.48
query35	4.55	4.48	4.57
query36	0.66	0.47	0.48
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.03	0.02
query40	0.16	0.13	0.13
query41	0.07	0.02	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.38 s
Total hot run time: 31.71 s

@morningman morningman force-pushed the fix-upgrade-paimon-3.0 branch from 028b5ec to 10bd080 Compare April 10, 2025 05:00
@morningman
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39971 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 10bd08087a30448d5f55a8e4c3be1c8f8c271fad, data reload: false

------ Round 1 ----------------------------------
q1	17599	6825	6574	6574
q2	2068	171	167	167
q3	10557	1111	1105	1105
q4	10588	789	675	675
q5	7733	2824	2837	2824
q6	219	135	129	129
q7	999	626	611	611
q8	9357	1966	2044	1966
q9	6597	6387	6372	6372
q10	6973	2299	2277	2277
q11	458	255	256	255
q12	413	208	208	208
q13	17764	3001	2972	2972
q14	228	212	216	212
q15	505	457	456	456
q16	664	573	582	573
q17	979	539	574	539
q18	7187	6687	6658	6658
q19	1402	1136	1075	1075
q20	467	201	199	199
q21	4038	3307	3170	3170
q22	1097	954	976	954
Total cold run time: 107892 ms
Total hot run time: 39971 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6601	6609	6542	6542
q2	332	234	242	234
q3	2953	2785	2898	2785
q4	2029	1814	1797	1797
q5	5749	5740	5719	5719
q6	211	132	127	127
q7	2196	1819	1809	1809
q8	3386	3535	3492	3492
q9	8884	8904	9076	8904
q10	3589	3551	3540	3540
q11	605	501	502	501
q12	864	637	626	626
q13	10479	3209	3171	3171
q14	314	271	272	271
q15	520	464	460	460
q16	704	660	636	636
q17	1841	1579	1624	1579
q18	8340	7737	7651	7651
q19	1680	1441	1461	1441
q20	2098	1811	1896	1811
q21	5374	5390	5282	5282
q22	1131	1027	1033	1027
Total cold run time: 69880 ms
Total hot run time: 59405 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197262 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 10bd08087a30448d5f55a8e4c3be1c8f8c271fad, data reload: false

query1	1294	898	888	888
query2	6242	2103	2060	2060
query3	10811	4145	4343	4145
query4	60795	33869	23756	23756
query5	5151	462	431	431
query6	312	190	166	166
query7	4901	315	322	315
query8	280	256	217	217
query9	6168	2628	2590	2590
query10	430	271	251	251
query11	15699	15213	15680	15213
query12	149	103	111	103
query13	1033	479	416	416
query14	10870	7736	6857	6857
query15	203	181	177	177
query16	7193	481	480	480
query17	1174	584	592	584
query18	1838	329	332	329
query19	199	161	162	161
query20	137	110	108	108
query21	213	107	110	107
query22	4679	4520	4733	4520
query23	34652	34089	34056	34056
query24	6177	2991	2936	2936
query25	537	456	429	429
query26	658	178	178	178
query27	1823	359	365	359
query28	3956	2477	2419	2419
query29	705	476	433	433
query30	241	162	172	162
query31	1017	830	814	814
query32	67	55	58	55
query33	429	306	333	306
query34	915	516	519	516
query35	838	721	735	721
query36	1097	968	996	968
query37	115	69	62	62
query38	4108	3915	4138	3915
query39	1530	1480	1470	1470
query40	206	99	98	98
query41	49	46	47	46
query42	115	102	98	98
query43	520	496	498	496
query44	1168	823	828	823
query45	182	173	167	167
query46	1129	715	725	715
query47	2051	1960	1988	1960
query48	463	381	402	381
query49	810	412	404	404
query50	836	419	413	413
query51	7336	7185	7359	7185
query52	101	88	90	88
query53	256	184	186	184
query54	579	461	454	454
query55	80	79	74	74
query56	279	264	250	250
query57	1301	1178	1111	1111
query58	222	204	215	204
query59	3230	3029	2937	2937
query60	278	250	242	242
query61	111	108	109	108
query62	777	687	688	687
query63	220	193	186	186
query64	1348	676	642	642
query65	3243	3153	3152	3152
query66	699	295	322	295
query67	16151	15659	15670	15659
query68	3907	577	572	572
query69	425	259	266	259
query70	1202	1160	1131	1131
query71	342	253	260	253
query72	6425	4022	4069	4022
query73	753	346	360	346
query74	10287	8967	8944	8944
query75	3381	2645	2643	2643
query76	1717	1081	1061	1061
query77	529	277	275	275
query78	10608	9681	9561	9561
query79	1744	594	603	594
query80	893	440	420	420
query81	504	243	234	234
query82	1265	89	84	84
query83	171	145	151	145
query84	283	81	82	81
query85	882	298	292	292
query86	335	300	303	300
query87	4514	4325	4324	4324
query88	3905	2383	2346	2346
query89	414	287	286	286
query90	1960	181	191	181
query91	184	150	151	150
query92	64	47	50	47
query93	2161	556	557	556
query94	730	294	301	294
query95	360	257	247	247
query96	616	283	277	277
query97	3339	3194	3143	3143
query98	208	217	195	195
query99	1565	1294	1267	1267
Total cold run time: 308049 ms
Total hot run time: 197262 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.4 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 10bd08087a30448d5f55a8e4c3be1c8f8c271fad, data reload: false

query1	0.03	0.03	0.02
query2	0.07	0.02	0.03
query3	0.24	0.07	0.07
query4	1.61	0.10	0.10
query5	0.52	0.52	0.51
query6	1.13	0.73	0.73
query7	0.02	0.01	0.01
query8	0.04	0.03	0.04
query9	0.57	0.50	0.51
query10	0.54	0.54	0.55
query11	0.14	0.10	0.11
query12	0.14	0.14	0.11
query13	0.61	0.60	0.60
query14	2.79	2.71	2.73
query15	0.90	0.83	0.84
query16	0.40	0.39	0.38
query17	1.07	1.06	1.06
query18	0.23	0.22	0.21
query19	1.93	1.84	2.04
query20	0.02	0.01	0.01
query21	15.36	0.58	0.55
query22	3.11	2.13	2.19
query23	16.83	1.09	0.84
query24	2.85	1.17	0.91
query25	0.24	0.05	0.06
query26	0.52	0.14	0.13
query27	0.05	0.04	0.03
query28	10.67	0.54	0.49
query29	12.57	3.31	3.25
query30	0.24	0.06	0.07
query31	2.85	0.40	0.38
query32	3.23	0.46	0.46
query33	2.96	3.02	3.05
query34	17.15	4.47	4.45
query35	4.55	4.47	4.53
query36	0.69	0.48	0.48
query37	0.09	0.06	0.07
query38	0.05	0.04	0.04
query39	0.04	0.03	0.02
query40	0.17	0.13	0.14
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.38 s
Total hot run time: 32.4 s

@morningman morningman force-pushed the fix-upgrade-paimon-3.0 branch from 10bd080 to d452b3c Compare April 19, 2025 23:05
@morningman
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39745 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d452b3cc5134d88e45c114243e7f4907ecf41c79, data reload: false

------ Round 1 ----------------------------------
q1	17633	6750	6594	6594
q2	2058	168	186	168
q3	10566	1078	1208	1078
q4	10569	729	771	729
q5	7734	2849	2821	2821
q6	220	138	134	134
q7	972	632	611	611
q8	9366	1944	2048	1944
q9	6554	6357	6366	6357
q10	7005	2231	2252	2231
q11	470	280	277	277
q12	402	217	214	214
q13	17781	2994	2982	2982
q14	233	209	208	208
q15	503	464	454	454
q16	661	585	606	585
q17	978	559	530	530
q18	7190	6486	6573	6486
q19	1394	1081	1023	1023
q20	479	203	211	203
q21	4018	3342	3138	3138
q22	1108	979	978	978
Total cold run time: 107894 ms
Total hot run time: 39745 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6577	6551	6893	6551
q2	338	235	226	226
q3	2906	2742	2845	2742
q4	2018	1803	1803	1803
q5	5801	5749	5701	5701
q6	214	130	132	130
q7	2195	1804	1798	1798
q8	3359	3589	3539	3539
q9	8965	8788	8996	8788
q10	3543	3520	3498	3498
q11	597	502	483	483
q12	808	612	635	612
q13	9335	3192	3191	3191
q14	307	267	268	267
q15	505	467	487	467
q16	716	661	649	649
q17	1838	1643	1628	1628
q18	8169	7758	7649	7649
q19	1711	1604	1647	1604
q20	2046	1801	1807	1801
q21	5665	5439	5422	5422
q22	1144	1030	1050	1030
Total cold run time: 68757 ms
Total hot run time: 59579 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197813 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d452b3cc5134d88e45c114243e7f4907ecf41c79, data reload: false

query1	1276	943	891	891
query2	6237	2023	1972	1972
query3	10966	4386	4376	4376
query4	61456	29618	23808	23808
query5	5187	483	454	454
query6	405	184	180	180
query7	5460	318	314	314
query8	305	238	226	226
query9	8637	2611	2564	2564
query10	457	278	263	263
query11	17408	15639	16024	15639
query12	158	103	106	103
query13	1461	464	428	428
query14	9904	7021	7272	7021
query15	197	183	179	179
query16	7164	528	496	496
query17	1134	583	608	583
query18	1842	337	307	307
query19	212	167	164	164
query20	120	117	113	113
query21	210	107	104	104
query22	4454	4480	4569	4480
query23	34193	34238	34119	34119
query24	6232	2960	2934	2934
query25	544	422	427	422
query26	674	181	176	176
query27	1947	367	384	367
query28	4109	2432	2448	2432
query29	707	485	445	445
query30	240	162	158	158
query31	937	820	839	820
query32	67	56	57	56
query33	461	282	279	279
query34	910	519	527	519
query35	854	741	722	722
query36	1074	952	943	943
query37	114	65	76	65
query38	4045	4030	3945	3945
query39	1529	1471	1452	1452
query40	202	99	108	99
query41	49	47	49	47
query42	111	100	99	99
query43	525	500	488	488
query44	1151	802	807	802
query45	183	167	169	167
query46	1150	751	737	737
query47	2016	1899	1950	1899
query48	493	397	386	386
query49	718	385	392	385
query50	838	451	445	445
query51	7389	7222	7431	7222
query52	99	92	92	92
query53	259	191	187	187
query54	581	476	485	476
query55	81	81	81	81
query56	266	260	248	248
query57	1243	1191	1153	1153
query58	225	218	228	218
query59	3159	3156	2952	2952
query60	282	253	249	249
query61	109	110	136	110
query62	775	678	667	667
query63	219	187	192	187
query64	1345	664	641	641
query65	3300	3204	3214	3204
query66	691	306	339	306
query67	15855	15700	15395	15395
query68	4178	604	580	580
query69	455	276	266	266
query70	1152	1150	1061	1061
query71	363	256	258	256
query72	6336	4004	4079	4004
query73	744	355	370	355
query74	10426	8915	9305	8915
query75	3340	2648	2662	2648
query76	1907	1006	1156	1006
query77	485	279	275	275
query78	10634	9627	9517	9517
query79	1717	603	620	603
query80	878	435	425	425
query81	514	237	235	235
query82	1265	93	92	92
query83	247	143	140	140
query84	279	81	80	80
query85	891	295	297	295
query86	322	294	302	294
query87	4392	4271	4197	4197
query88	3723	2429	2404	2404
query89	417	287	290	287
query90	2013	188	189	188
query91	179	154	151	151
query92	66	50	50	50
query93	1981	567	559	559
query94	763	285	299	285
query95	350	265	259	259
query96	619	279	286	279
query97	3271	3164	3143	3143
query98	218	202	196	196
query99	1598	1331	1307	1307
Total cold run time: 312324 ms
Total hot run time: 197813 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.75 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d452b3cc5134d88e45c114243e7f4907ecf41c79, data reload: false

query1	0.03	0.03	0.02
query2	0.06	0.03	0.03
query3	0.24	0.07	0.07
query4	1.63	0.10	0.10
query5	0.53	0.50	0.51
query6	1.14	0.72	0.72
query7	0.04	0.01	0.02
query8	0.04	0.03	0.03
query9	0.57	0.50	0.50
query10	0.54	0.54	0.53
query11	0.14	0.10	0.11
query12	0.14	0.11	0.12
query13	0.63	0.60	0.59
query14	2.76	2.78	2.84
query15	0.92	0.84	0.83
query16	0.39	0.40	0.37
query17	1.02	1.01	1.05
query18	0.24	0.22	0.23
query19	1.88	1.80	1.89
query20	0.01	0.02	0.01
query21	15.37	0.60	0.60
query22	2.61	2.73	2.40
query23	16.88	1.08	0.98
query24	3.36	1.84	0.66
query25	0.45	0.14	0.17
query26	0.29	0.13	0.14
query27	0.04	0.04	0.06
query28	9.71	0.56	0.51
query29	12.56	3.32	3.19
query30	0.25	0.06	0.05
query31	2.88	0.40	0.39
query32	3.23	0.46	0.46
query33	2.99	3.05	3.01
query34	17.13	4.52	4.56
query35	4.57	4.56	4.59
query36	0.64	0.47	0.49
query37	0.09	0.07	0.06
query38	0.04	0.03	0.04
query39	0.04	0.02	0.03
query40	0.16	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.39 s
Total hot run time: 32.75 s

@morningman morningman force-pushed the fix-upgrade-paimon-3.0 branch from d452b3c to 4459685 Compare April 22, 2025 05:43
@morningman
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40190 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 44596852a56b6ae62a51f124662a996fa5bab41b, data reload: false

------ Round 1 ----------------------------------
q1	17595	6783	6605	6605
q2	2097	177	168	168
q3	10602	1087	1167	1087
q4	10565	738	709	709
q5	7761	2885	2822	2822
q6	220	137	132	132
q7	980	621	610	610
q8	9355	1973	2034	1973
q9	6605	6405	6361	6361
q10	6995	2305	2334	2305
q11	464	264	266	264
q12	406	222	218	218
q13	17789	2967	2996	2967
q14	230	208	207	207
q15	506	462	459	459
q16	635	580	579	579
q17	966	534	546	534
q18	7164	6672	6817	6672
q19	1704	1039	1115	1039
q20	478	210	214	210
q21	4275	3355	3260	3260
q22	1139	1024	1009	1009
Total cold run time: 108531 ms
Total hot run time: 40190 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6699	6623	6794	6623
q2	324	228	233	228
q3	2915	2752	2940	2752
q4	2052	1835	1777	1777
q5	5785	5740	5776	5740
q6	202	131	127	127
q7	2202	1797	1789	1789
q8	3365	3537	3493	3493
q9	8809	8918	8918	8918
q10	3547	3556	3538	3538
q11	596	503	503	503
q12	809	645	640	640
q13	8005	3193	3179	3179
q14	302	265	267	265
q15	513	461	464	461
q16	708	633	665	633
q17	1851	1633	1666	1633
q18	8142	7803	7911	7803
q19	1710	1626	1603	1603
q20	2024	1779	1839	1779
q21	5536	5301	5330	5301
q22	1119	1053	1045	1045
Total cold run time: 67215 ms
Total hot run time: 59830 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197855 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 44596852a56b6ae62a51f124662a996fa5bab41b, data reload: false

query1	1255	895	906	895
query2	6240	2023	1990	1990
query3	10927	4398	4492	4398
query4	61484	29480	23913	23913
query5	5171	473	468	468
query6	406	189	181	181
query7	5469	323	338	323
query8	317	238	232	232
query9	8482	2613	2611	2611
query10	481	291	277	277
query11	17648	15145	15653	15145
query12	166	108	102	102
query13	1432	444	452	444
query14	9908	7051	6904	6904
query15	211	184	176	176
query16	7168	495	509	495
query17	1180	592	626	592
query18	1873	326	333	326
query19	219	164	163	163
query20	118	109	109	109
query21	202	101	113	101
query22	4660	4452	4524	4452
query23	34778	33839	33975	33839
query24	6846	3016	2985	2985
query25	545	423	424	423
query26	654	176	168	168
query27	2139	369	372	369
query28	4451	2464	2438	2438
query29	728	482	451	451
query30	245	160	164	160
query31	1009	822	837	822
query32	69	54	60	54
query33	450	289	288	288
query34	916	515	526	515
query35	881	733	753	733
query36	1068	945	963	945
query37	117	66	77	66
query38	4106	3970	4008	3970
query39	1517	1466	1472	1466
query40	199	101	101	101
query41	49	48	47	47
query42	114	105	105	105
query43	545	495	490	490
query44	1220	814	837	814
query45	185	169	167	167
query46	1161	735	754	735
query47	2025	1961	1980	1961
query48	485	400	404	400
query49	725	397	405	397
query50	857	440	428	428
query51	7473	7160	7355	7160
query52	96	88	90	88
query53	275	195	189	189
query54	602	479	460	460
query55	81	79	82	79
query56	251	257	252	252
query57	1262	1158	1178	1158
query58	222	224	219	219
query59	3180	3067	2955	2955
query60	282	269	266	266
query61	116	130	182	130
query62	800	676	690	676
query63	221	194	196	194
query64	1406	722	677	677
query65	3295	3216	3210	3210
query66	716	307	308	307
query67	15938	15689	15489	15489
query68	4404	589	574	574
query69	449	276	267	267
query70	1151	1148	1125	1125
query71	361	257	259	257
query72	6420	3985	4014	3985
query73	793	339	349	339
query74	10145	9199	9151	9151
query75	3369	2680	2655	2655
query76	1940	1048	1111	1048
query77	540	278	277	277
query78	10659	9628	9600	9600
query79	1741	602	607	602
query80	889	449	435	435
query81	532	239	237	237
query82	1270	94	90	90
query83	254	147	143	143
query84	284	86	82	82
query85	906	314	298	298
query86	342	304	271	271
query87	4441	4200	4338	4200
query88	3715	2488	2393	2393
query89	430	299	297	297
query90	1976	189	187	187
query91	187	151	146	146
query92	60	50	50	50
query93	1990	569	563	563
query94	763	302	304	302
query95	367	270	256	256
query96	619	279	283	279
query97	3291	3159	3120	3120
query98	218	208	206	206
query99	1546	1277	1283	1277
Total cold run time: 315369 ms
Total hot run time: 197855 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.84 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 44596852a56b6ae62a51f124662a996fa5bab41b, data reload: false

query1	0.03	0.04	0.03
query2	0.06	0.03	0.03
query3	0.23	0.07	0.07
query4	1.62	0.10	0.10
query5	0.54	0.51	0.51
query6	1.13	0.73	0.74
query7	0.02	0.01	0.01
query8	0.04	0.05	0.02
query9	0.56	0.51	0.50
query10	0.55	0.55	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.60	0.60	0.59
query14	2.70	2.82	2.76
query15	0.89	0.81	0.84
query16	0.40	0.38	0.37
query17	1.05	1.02	1.07
query18	0.24	0.22	0.22
query19	1.93	1.73	2.04
query20	0.02	0.01	0.00
query21	15.36	0.59	0.58
query22	2.45	2.89	1.66
query23	17.03	0.96	0.79
query24	2.97	1.68	0.75
query25	0.12	0.12	0.14
query26	0.60	0.13	0.14
query27	0.05	0.05	0.04
query28	10.20	0.50	0.50
query29	12.62	3.25	3.27
query30	0.24	0.06	0.06
query31	2.86	0.39	0.39
query32	3.22	0.47	0.45
query33	2.99	3.03	3.03
query34	16.86	4.54	4.57
query35	4.57	4.53	4.54
query36	0.66	0.47	0.48
query37	0.08	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.03	0.02
query40	0.16	0.13	0.12
query41	0.08	0.02	0.03
query42	0.04	0.02	0.02
query43	0.03	0.03	0.02
Total cold run time: 106.15 s
Total hot run time: 31.84 s

wuwenchi and others added 2 commits April 24, 2025 15:21
@wuwenchi
[opt](paimon)Upgrade the Paimon version to 1.0.0 and Iceberg to 1.6.1 (…
9ba4123 
…apache#46990)

Problem Summary:

Upgrade the Paimon version to 1.0.0

By default, paimon uses a caching catalog to cache some data to improve
read performance.
FYI:
https://paimon.apache.org/docs/1.0/maintenance/configurations/#catalogoptions

If you do not want to use this catalog, you can add a configuration
`paimon.cache-enabled ` to turn it off:
```
CREATE CATALOG `c1` PROPERTIES (
"type" = "paimon",
"paimon.catalog.type" = "xxx",
"paimon.cache-enabled" = "false",
"warehouse" = "xxx"
);
```
If you want to modify cache-related parameters, you can add the
`paimon.` prefix to the parameters supported by paimon, such as:
```
CREATE CATALOG `c1` PROPERTIES (
"type" = "paimon",
"paimon.catalog.type" = "xxx",
"warehouse" = "xxx",
"paimon.cache.expiration-interval" = "20 min",
"paimon.cache.manifest.small-file-memory"="10 mb"
);
```

Note:
During the doris upgrade process, this error may occur:

![image](https://github.com/user-attachments/assets/47ec8216-9e3f-4d8e-95ef-17cce6b7c486)

This is because doris will upgrade be first, and then upgrade fe. During
this process, the version of paimon on be may be higher than that on fe.
This is normal. Because bucketkey judgment is newly added in the higher
version of paimon, which is not available in the lower version. After
the fe upgrade is completed normally, there will be no more errors.
@suxiaogang223 @wuwenchi
[upgrade](paimon) upgrade paimon 1.0.1 (apache#47768)
0d5185b 
### Release note

Upgrade paimon to version 1.0.1
@wuwenchi wuwenchi force-pushed the fix-upgrade-paimon-3.0 branch from 4459685 to 0d5185b Compare April 24, 2025 07:22
@wuwenchi
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39861 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0d5185b02d8aec082c662090031027b40a9870c3, data reload: false

------ Round 1 ----------------------------------
q1	17578	6739	6563	6563
q2	2063	167	161	161
q3	10725	1054	1220	1054
q4	10518	699	742	699
q5	7705	2820	2773	2773
q6	217	132	128	128
q7	955	604	602	602
q8	9480	1981	1973	1973
q9	6708	6479	6468	6468
q10	7139	2276	2318	2276
q11	467	261	256	256
q12	400	218	213	213
q13	17779	2930	2977	2930
q14	240	208	210	208
q15	503	464	460	460
q16	662	605	592	592
q17	969	604	644	604
q18	7260	6690	6593	6593
q19	1397	1047	1134	1047
q20	459	204	198	198
q21	3921	3086	3220	3086
q22	1108	989	977	977
Total cold run time: 108253 ms
Total hot run time: 39861 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6647	6550	6540	6540
q2	329	231	229	229
q3	2903	2802	2919	2802
q4	2041	1756	1771	1756
q5	5718	5697	5682	5682
q6	226	127	129	127
q7	2220	1779	1790	1779
q8	3322	3531	3491	3491
q9	8795	8795	8807	8795
q10	3565	3500	3459	3459
q11	594	489	485	485
q12	798	581	624	581
q13	8007	3201	3130	3130
q14	320	271	288	271
q15	507	462	472	462
q16	678	648	640	640
q17	1827	1608	1589	1589
q18	8140	7873	7768	7768
q19	1692	1569	1523	1523
q20	2048	1868	1866	1866
q21	5560	5350	5192	5192
q22	1148	1049	1036	1036
Total cold run time: 67085 ms
Total hot run time: 59203 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197620 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0d5185b02d8aec082c662090031027b40a9870c3, data reload: false

query1	1274	911	907	907
query2	6354	2051	2030	2030
query3	10774	4387	4158	4158
query4	61275	29556	23673	23673
query5	5204	465	452	452
query6	413	178	183	178
query7	5428	320	310	310
query8	306	230	224	224
query9	8588	2644	2622	2622
query10	446	289	253	253
query11	17880	15215	15790	15215
query12	169	105	115	105
query13	1401	451	423	423
query14	10046	7778	7152	7152
query15	197	178	182	178
query16	7124	493	468	468
query17	1184	572	596	572
query18	1901	325	340	325
query19	214	160	160	160
query20	115	112	108	108
query21	210	106	102	102
query22	4552	4388	4584	4388
query23	34651	34078	34140	34078
query24	6162	2941	2950	2941
query25	555	423	422	422
query26	671	178	174	174
query27	1876	362	349	349
query28	4261	2508	2436	2436
query29	702	485	469	469
query30	245	166	168	166
query31	986	825	851	825
query32	72	60	63	60
query33	410	300	303	300
query34	922	534	538	534
query35	834	749	745	745
query36	1087	936	961	936
query37	124	75	73	73
query38	4087	4022	4057	4022
query39	1561	1480	1476	1476
query40	200	100	97	97
query41	48	49	48	48
query42	117	100	100	100
query43	535	484	489	484
query44	1138	811	821	811
query45	187	171	176	171
query46	1175	731	717	717
query47	2001	1931	1970	1931
query48	499	405	385	385
query49	704	417	397	397
query50	842	431	421	421
query51	7240	7252	7282	7252
query52	104	98	101	98
query53	265	190	187	187
query54	590	480	474	474
query55	84	82	82	82
query56	266	248	239	239
query57	1259	1144	1147	1144
query58	224	215	211	211
query59	3211	3096	3019	3019
query60	291	265	249	249
query61	110	108	109	108
query62	769	672	673	672
query63	218	182	186	182
query64	1447	696	630	630
query65	3262	3197	3186	3186
query66	644	307	301	301
query67	15704	15711	15524	15524
query68	4155	588	582	582
query69	431	272	284	272
query70	1190	1145	1113	1113
query71	318	264	253	253
query72	6346	4142	4048	4048
query73	746	358	358	358
query74	10062	8972	9171	8972
query75	3340	2651	2616	2616
query76	2101	1073	983	983
query77	477	268	276	268
query78	10928	9700	9632	9632
query79	1968	604	605	604
query80	1332	432	433	432
query81	527	237	234	234
query82	1098	90	87	87
query83	274	141	138	138
query84	287	76	81	76
query85	1047	298	290	290
query86	378	286	277	277
query87	4395	4205	4243	4205
query88	3881	2417	2367	2367
query89	415	301	292	292
query90	1894	191	188	188
query91	183	148	151	148
query92	66	50	52	50
query93	2659	572	565	565
query94	740	295	309	295
query95	355	266	261	261
query96	611	284	283	283
query97	3296	3137	3124	3124
query98	215	199	194	194
query99	1636	1284	1297	1284
Total cold run time: 314746 ms
Total hot run time: 197620 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.35 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0d5185b02d8aec082c662090031027b40a9870c3, data reload: false

query1	0.04	0.04	0.03
query2	0.06	0.03	0.03
query3	0.23	0.06	0.06
query4	1.63	0.10	0.10
query5	0.53	0.51	0.53
query6	1.14	0.74	0.72
query7	0.02	0.02	0.01
query8	0.03	0.03	0.03
query9	0.56	0.49	0.50
query10	0.53	0.54	0.55
query11	0.15	0.11	0.11
query12	0.13	0.11	0.11
query13	0.62	0.60	0.60
query14	2.87	2.73	2.79
query15	0.90	0.82	0.82
query16	0.37	0.38	0.36
query17	1.03	1.05	1.10
query18	0.24	0.23	0.22
query19	1.96	1.83	2.04
query20	0.02	0.00	0.01
query21	15.37	0.61	0.61
query22	2.44	2.36	2.46
query23	16.99	0.88	0.83
query24	3.65	0.57	1.34
query25	0.22	0.19	0.14
query26	0.44	0.14	0.14
query27	0.04	0.03	0.03
query28	10.04	0.52	0.44
query29	12.69	3.26	3.24
query30	0.25	0.06	0.07
query31	2.85	0.40	0.39
query32	3.23	0.46	0.46
query33	2.93	2.99	3.00
query34	16.81	4.54	4.47
query35	4.56	4.59	4.49
query36	0.66	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.04	0.02	0.03
query40	0.16	0.13	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.72 s
Total hot run time: 32.35 s

@morningman @wuwenchi
[opt](mvn) remove iceberg-aws-bundle and upgrade aws sdk to 2.29.26 (a…
9790fe2 
…pache#47117)

### What problem does this PR solve?

1. No need to depend on iceberg-aws-bundle, because we've already
declare the depends we need separately.
2. upgrade aws sdk to 2.29.26 to prepare for s3 tables
@wuwenchi
Copy link
Contributor Author

run buildall

dataroaring
dataroaring approved these changes Apr 25, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit a9da4de into apache:branch-3.0 Apr 25, 2025
20 of 22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@wuwenchi @hello-stephen @doris-robot @morningman @dataroaring @suxiaogang223