The architecture of Loader

[h-a-n-a]

Related PRs

• feat: impl new loader-runner for preparing to support vue-loader and pitching loader #2789
• feat: support loader-runner and pitching #2808

Summary

Cited from the message of PR.

This PR refactors the loader architecture and adapts webpack's loader-runner to rspack.
Pitching loader is supported for both Rust and JS side from now on.

The old architecture is a quite simple version, which only supports loaders for normal stage.
Pitching loader does not put into consideration. The basic concept of the old version is to
convert the normal loader to a native function which can be called from the Rust side.
Furthermore, for performance reason, Rspack also composes loaders from the JS side to
mitigate the performance issue of Node/Rust communications.

In this new architecture, loaders will not be converted directly into native functions.
Instead, it is almost the same with how webpack's loader-runner resolves its loaders, by
leveraging the identifier. Every time Rspack wants to invoke a JS loader, the identifiers will
be passed to the handler passed by Node side to process. The implementation also keeps
the feature of composing JS loaders for performance reason.

Guide-level explanation

The refactor does not introduce any other breaking changes. So it's backwards compatible.
The change of the architecture also help us to implement pitching loader with composability.

Pitching loader

Pitching loader is a technique to change the loader pipeline flow. It is usually used with
inline loader syntax for creating another loader pipeline. style-loader, etc and other loaders
which might consume the evaluated result of the following loaders may use this technique.
There are other technique to achieve the same ability, but it's out of this article's topic.

See Pitching loader for more detail.

Reference-level explanation

Actor of loader execution

In the original implementation of loader, Rspack will convert the normal loaders in the first place,
then pass it to the Rust side. In the procedure of building modules, these loaders will be called directly:

The loader runner is only on the Rust side and execute the loaders directly from the Rust side.
This mechanism has a strong limit for us to use webpack's loader-runner for composed loaders.

In the new architecture, we will delegate the loader request from the Rust core to a dispatcher
located on the JS side. The dispatcher will normalize the loader and execute these using a modified
version of webpack's loader-runner:

Loader functions for pitch or normal will not be passed to the Rust side. Instead, each JS loader has
its identifier to uniquely represent each one. If a module requests a loader for processing the module,
Rspack will pass identifier with options to the JS side to instruct the Webpack like loader-runner to
process the transform. This also reduces the complexity of writing our own loader composer.

Passing options

Options will normally be converted to query, but some of the options contain fields that cannot be
serialized, Rspack will reuse the loader ident created by webpack to uniquely identify the option
and restore it in later loading process.

Optimization for pitching

As we had known before, each loader has two steps, pitch and normal. For a performance friendly
interoperability, we must reduce the communication between Rust and JS as minimum as possible.
Normally, the execution steps of loaders will look like this:

The execution order of the loaders above will looks like this:

loader-A(pitch)
   loader-B(pitch)
      loader-C(pitch)
   loader-B(normal)
loader-A(normal)

The example above does not contain any JS loaders, but if, say, we mark these loaders registered on the
JS side:

The execution order will not change, but Rspack will compose the step 2/3/4 together for only a single
round communication.