Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](exchange)fix exchange sink buffer does not update total_queue_size when EOF. #47312

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

Mryange
Copy link
Contributor

@Mryange Mryange commented Jan 22, 2025

What problem does this PR solve?

#41602
EOF clears _instance_to_package_queue but does not update total_queue_size, causing incorrect judgments that rely on total_queue_size.

UT

mock transmit_blockv2 dest ins id :1
mock transmit_blockv2 dest ins id :2
mock transmit_blockv2 dest ins id :3
queue size : 6
each queue size : 
Instance: 2, queue size: 2
Instance: 1, queue size: 2
Instance: 3, queue size: 2

queue size : 6 // error size 
each queue size :
Instance: 2, queue size: 0
Instance: 1, queue size: 2
Instance: 3, queue size: 2

mock transmit_blockv2 dest ins id :1
mock transmit_blockv2 dest ins id :1
mock transmit_blockv2 dest ins id :3
mock transmit_blockv2 dest ins id :3

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jan 22, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Mryange
Copy link
Contributor Author

Mryange commented Jan 22, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32405 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 294970dee21037f3646502800971608eb4f60787, data reload: false

------ Round 1 ----------------------------------
q1	17592	5546	5409	5409
q2	2065	302	173	173
q3	10494	1216	744	744
q4	10223	983	547	547
q5	7646	2398	2191	2191
q6	198	174	133	133
q7	910	761	598	598
q8	9237	1358	1192	1192
q9	5153	4918	4975	4918
q10	6842	2331	1891	1891
q11	488	270	260	260
q12	345	360	224	224
q13	17802	3732	3081	3081
q14	244	233	215	215
q15	533	473	465	465
q16	619	608	595	595
q17	579	869	333	333
q18	7126	6676	6411	6411
q19	2893	958	521	521
q20	307	320	195	195
q21	2823	2224	1995	1995
q22	359	339	314	314
Total cold run time: 104478 ms
Total hot run time: 32405 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5594	5518	5536	5518
q2	238	342	244	244
q3	2284	2675	2362	2362
q4	1444	1832	1378	1378
q5	4325	4768	4783	4768
q6	174	164	125	125
q7	2048	1986	1845	1845
q8	2654	2832	2753	2753
q9	7387	7304	7364	7304
q10	3018	3299	2700	2700
q11	563	505	491	491
q12	646	774	637	637
q13	3592	3895	3275	3275
q14	286	307	285	285
q15	510	498	463	463
q16	651	696	653	653
q17	1230	1755	1257	1257
q18	7742	7551	7379	7379
q19	806	883	1210	883
q20	1993	2067	1878	1878
q21	5752	5056	4990	4990
q22	603	599	587	587
Total cold run time: 53540 ms
Total hot run time: 51775 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 194815 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 294970dee21037f3646502800971608eb4f60787, data reload: false

query1	1326	972	927	927
query2	6283	2051	1959	1959
query3	11158	4641	4719	4641
query4	60943	28992	23508	23508
query5	5470	589	456	456
query6	411	194	174	174
query7	5536	507	299	299
query8	315	237	236	236
query9	8155	2578	2559	2559
query10	469	317	252	252
query11	17459	15375	15374	15374
query12	173	124	113	113
query13	1470	548	413	413
query14	11512	6951	6789	6789
query15	222	208	187	187
query16	7204	661	467	467
query17	1180	761	583	583
query18	1870	417	323	323
query19	218	186	162	162
query20	121	117	115	115
query21	211	125	103	103
query22	4732	4559	4571	4559
query23	34124	33805	33361	33361
query24	5612	2362	2401	2362
query25	461	492	394	394
query26	661	238	151	151
query27	1792	476	333	333
query28	4325	2468	2448	2448
query29	563	563	425	425
query30	213	208	155	155
query31	925	880	826	826
query32	66	62	55	55
query33	421	375	304	304
query34	733	876	494	494
query35	808	862	767	767
query36	1021	1060	965	965
query37	130	105	92	92
query38	4257	4392	4244	4244
query39	1503	1443	1433	1433
query40	208	141	104	104
query41	53	50	52	50
query42	117	103	101	101
query43	521	517	492	492
query44	1305	805	814	805
query45	191	176	173	173
query46	883	1074	669	669
query47	1949	1954	1894	1894
query48	384	415	319	319
query49	723	501	400	400
query50	668	670	409	409
query51	7147	7203	6987	6987
query52	100	98	97	97
query53	224	253	183	183
query54	491	516	445	445
query55	85	77	105	77
query56	273	268	261	261
query57	1202	1206	1133	1133
query58	249	246	231	231
query59	3080	3008	2831	2831
query60	284	261	247	247
query61	139	113	131	113
query62	709	723	638	638
query63	221	183	187	183
query64	1332	1033	721	721
query65	3224	3151	3185	3151
query66	708	400	290	290
query67	16134	15695	15499	15499
query68	5004	832	521	521
query69	483	302	266	266
query70	1176	1103	1098	1098
query71	412	283	255	255
query72	6029	3969	3835	3835
query73	804	752	358	358
query74	9786	8839	8862	8839
query75	3207	3174	2627	2627
query76	3750	1185	790	790
query77	469	371	289	289
query78	10057	9957	9452	9452
query79	2532	814	598	598
query80	1284	540	453	453
query81	529	295	242	242
query82	356	156	119	119
query83	253	172	160	160
query84	291	97	74	74
query85	818	350	321	321
query86	406	318	305	305
query87	4528	4450	4302	4302
query88	3897	2191	2156	2156
query89	404	331	293	293
query90	1606	191	201	191
query91	136	138	107	107
query92	67	56	53	53
query93	2879	887	535	535
query94	810	416	299	299
query95	333	276	267	267
query96	486	623	278	278
query97	2800	2862	2746	2746
query98	220	200	196	196
query99	1278	1359	1279	1279
Total cold run time: 313747 ms
Total hot run time: 194815 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.43 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 294970dee21037f3646502800971608eb4f60787, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.07
query4	1.61	0.11	0.10
query5	0.43	0.42	0.40
query6	1.16	0.65	0.67
query7	0.03	0.02	0.01
query8	0.04	0.03	0.03
query9	0.59	0.50	0.49
query10	0.57	0.57	0.56
query11	0.15	0.10	0.10
query12	0.14	0.10	0.11
query13	0.61	0.60	0.61
query14	2.75	2.82	2.72
query15	0.90	0.83	0.83
query16	0.37	0.40	0.38
query17	1.00	1.03	0.99
query18	0.25	0.22	0.21
query19	1.88	1.73	1.98
query20	0.02	0.02	0.01
query21	15.36	0.88	0.58
query22	0.77	0.76	0.69
query23	15.24	1.44	0.59
query24	3.02	1.51	0.73
query25	0.19	0.09	0.20
query26	0.38	0.15	0.14
query27	0.05	0.05	0.04
query28	13.68	1.04	0.44
query29	12.62	3.88	3.25
query30	0.25	0.09	0.06
query31	2.83	0.61	0.38
query32	3.22	0.56	0.46
query33	2.98	3.09	3.03
query34	16.56	5.11	4.53
query35	4.58	4.64	4.58
query36	0.66	0.52	0.49
query37	0.10	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.14	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.73 s
Total hot run time: 30.43 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 41.56% (10835/26070)
Line Coverage: 31.96% (91585/286552)
Region Coverage: 31.10% (46910/150852)
Branch Coverage: 27.19% (23764/87394)
Coverage Report: http://coverage.selectdb-in.cc/coverage/294970dee21037f3646502800971608eb4f60787_294970dee21037f3646502800971608eb4f60787/report/index.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants