RFC-025: Test Infrastructure

[Boshen]

At the moment of writing, our test infrastructure need to be improved. We are spending way too much time doing ad-hoc debugging.

I'll use structured comments for gathering feed backs.

--

Overall Conclusion:

Need a guide.

[Boshen]

Unit Tests

• some / a little
• cargo test - 64 tests

Conclusion

Unit tests are fine as is, people can freely add them in the Rust source whenever they feel appropriate.

[Boshen]

End-to-End

• no e2e
• no hmr - test websocket

Conclusion

This is nice to have, not top priority right now.

[Boshen]

Smoke Tests

• have build example test, need to fix exit code if the rspack build is broken
  • both internal and external example repos and folders

[Boshen]

Testing Business Partner Applications

• manual
• expect there is a already setup repo for rspack to test our nightly / snapshot
• e.g. vite ecosystem ci
  • integrate with modern.js's integration tests
• Need for minimal repro

Conclusion

We need at least something, no final conclusion yet.

[Boshen]

Prevent Performance Degradation

• no performance for individual loader / plugins / core algorithm
  • see webpack - can get individual bench results
• add more examples
• upload stats

Conclusion

• We should setup benchmark so it reports performance of individual components.
• We should upload and watch for performance degration for our internal business partners.

[Boshen]

Snapshots

• cargo rst to update snapshots - only some team members are using this?
• pnpm jest
  • not great for dx, because we have to build the project
• Migrate the test suite of Webpack #1732
  its ok to port manually, but we need to look at the gap / difference
• https://github.com/modern-js-dev/rspack/tree/main/packages/rspack/tests
• problem: webpack / rspack tests are mixed
• too much noise

Conclusion

This is currently a mess, but we should at least have a consensus on what to snapshot when we add feature.

Further discussions needed.

[Boshen]

Rust Integration Tests

• Move all to rspack crate
• Clean up https://github.com/modern-js-dev/rspack/tree/main/crates/rspack/tests/fixtures

Conclusion

We should clean this up.

[Boshen]

JS Integration Tests

• https://github.com/modern-js-dev/rspack/tree/main/packages/rspack/tests
• hard to debug
• can't test in isolation, need --filter

Conclusion

This is our preferred way of testing. We should improve our toolings for better developer experience.

We also need a way to compare our test suite against webpack for making visibility on our compatibility with webpack.

[hyf0]

It just came to me. I believe the key is to reduce complexity. IMO, we should remove all integration tests in Rust, which leads us to no more test.config.json. So, we don't need to maintain the second options transforming logic in Rust. We could remove much code that just does transforming.

With a good design of the test system in js, it might be enough for testing correctness.

I personally prefer testing in Rust. But It seems a mess in practice for such a project.

[Boshen]

I'll come up with a plan after we go open source, doing this right now makes too much of a burden on all of us.