refactor: introducing the faster code-splitting algorithm

[JSerFeng]

Summary

Change the algorithm of code-splitting.

Making it more maintainable, concurrent and easy to working with incoming incremental arch.

Because of the multi-threads, now the code-splitting is more performant than before even using single thread. There can be a huge improvement on large repo.

There is a internal test project which contains about 1.5 million chunk modules, using many dynamic imports, costs about 1200 ms to complete a whole code-splitting phase, using the new algorithm on my 15 core MacBook Pro M1, only takse 600 ms to finish !

The detailed benchmark data will be present soon

Checklist

• Tests updated (or not required).
• Documentation updated (or not required).

[netlify]

✅ Deploy Preview for rspack canceled.

Built without sensitive environment variables

Name	Link
🔨 Latest commit	e76cafc
🔍 Latest deploy log	https://app.netlify.com/sites/rspack/deploys/678a26839a44330008b43780

[codspeed-hq]

CodSpeed Performance Report

Merging #8823 will not alter performance

_{Comparing JSerFeng:refactor/code-splitter (e76cafc) with main (8d6406d)}

Summary

✅ 3 untouched benchmarks

[hardfist]

maybe we should add a benchmark case with lots of dynamic import module first and to see the bench metrics avoid future regression

[github-actions]

📝 Benchmark detail: Open

Name	Base (2025-01-06 f81f1c7)	Current	Change
10000_big_production-mode_disable-minimize + exec	38 s ± 659 ms	38 s ± 421 ms	+0.02 %
10000_development-mode + exec	1.91 s ± 39 ms	1.88 s ± 32 ms	-1.57 %
10000_development-mode_hmr + exec	688 ms ± 25 ms	676 ms ± 27 ms	-1.73 %
10000_production-mode + exec	2.51 s ± 29 ms	2.54 s ± 88 ms	+1.12 %
arco-pro_development-mode + exec	1.78 s ± 139 ms	1.75 s ± 62 ms	-1.71 %
arco-pro_development-mode_hmr + exec	378 ms ± 4.1 ms	378 ms ± 1.7 ms	-0.05 %
arco-pro_production-mode + exec	3.61 s ± 65 ms	3.53 s ± 99 ms	-2.27 %
arco-pro_production-mode_generate-package-json-webpack-plugin + exec	3.68 s ± 100 ms	3.67 s ± 84 ms	-0.27 %
arco-pro_production-mode_traverse-chunk-modules + exec	3.64 s ± 80 ms	3.59 s ± 109 ms	-1.29 %
threejs_development-mode_10x + exec	1.5 s ± 15 ms	1.5 s ± 12 ms	-0.25 %
threejs_development-mode_10x_hmr + exec	771 ms ± 14 ms	765 ms ± 21 ms	-0.75 %
threejs_production-mode_10x + exec	5.36 s ± 68 ms	5.38 s ± 157 ms	+0.45 %
10000_big_production-mode_disable-minimize + rss memory	9499 MiB ± 279 MiB	9571 MiB ± 141 MiB	+0.76 %
10000_development-mode + rss memory	666 MiB ± 20.7 MiB	653 MiB ± 22.6 MiB	-1.97 %
10000_development-mode_hmr + rss memory	1433 MiB ± 386 MiB	1392 MiB ± 191 MiB	-2.87 %
10000_production-mode + rss memory	623 MiB ± 22.8 MiB	628 MiB ± 18.4 MiB	+0.92 %
arco-pro_development-mode + rss memory	567 MiB ± 29.5 MiB	559 MiB ± 35.5 MiB	-1.25 %
arco-pro_development-mode_hmr + rss memory	625 MiB ± 64.7 MiB	598 MiB ± 56.6 MiB	-4.36 %
arco-pro_production-mode + rss memory	722 MiB ± 54.8 MiB	702 MiB ± 65.9 MiB	-2.73 %
arco-pro_production-mode_generate-package-json-webpack-plugin + rss memory	712 MiB ± 54.5 MiB	713 MiB ± 60 MiB	+0.08 %
arco-pro_production-mode_traverse-chunk-modules + rss memory	725 MiB ± 62.4 MiB	713 MiB ± 51.2 MiB	-1.61 %
threejs_development-mode_10x + rss memory	574 MiB ± 11.9 MiB	569 MiB ± 18.1 MiB	-0.91 %
threejs_development-mode_10x_hmr + rss memory	1141 MiB ± 70.6 MiB	1118 MiB ± 198 MiB	-2.02 %
threejs_production-mode_10x + rss memory	842 MiB ± 37.9 MiB	837 MiB ± 39.1 MiB	-0.52 %