RFC-003: Dependency abstraction

[h-a-n-a]

THIS RFC is withdrawn in favor of the WIP dependency RFC.

• Start Date: 2022-07-08
• Related RFCs: RFC-002: Asset support #405

Summary

This RFC proposes adding a dependency abstraction. Adding this abstraction will help rspack to add these functionalities:

• replace any import statement with, for example, inline-base64 statement, and point the reference to the generated location of each asset
• import assertions support

Motivation

For the status quo, after the asset support, rspack can handle the asset emitting and generate a javascript-based module like this:

Importing asset from JavaScript-like modules

import img from "./assets/asset.png"

- src
    | - assets/asset.png
    | - index.js

which generates,

// index.js
rs.define("./assets/asset.png", (module) => {
  module.exports = "assets/asset.png"
})

// entry module is omitted

- dist
    | - assets/asset.png
    | - index.js

Support for CSS

Given an example for CSS, you can found the solution above can only resolve part of the problem.

body {
    background-image: url("./images/file.png");
}

Since an asset can be formatted as AssetResource / AssetInline, etc. The canonicalized url function might be changed base on its max-size settings, which means the content of the url might be replaced. The reason why this wouldn't affect JavaScript-like modules is that we wrap JavaScript modules into runtimes which can return a new url or inlined based64, however, we cannot take the advantage of this for CSS, thus, Dependency comes for solving this kind of problems:

body {
   background-image: url("replaced-url");
}

Affected examples

Here are some additional examples that require dependency abstraction.

CSSUrlDependency

body {
  background-image: url("./images/file.png");
}

URL should be pointed to the asset location or inlined.

CSSImportDependency

@import "https://cdn/file.css";
/* note the `url` part is belonged to the media query part of the import dependency */
@import url("https://cdn/file.css");

CSS imports with media queries / supports query

@import url("fineprint.css") print;
@import url("bluish.css") print, screen;
@import "common.css" screen;
@import url('landscape.css') screen and (orientation: landscape);
@import url("narrow.css") supports(display: flex) screen and (max-width: 400px);

External resources are intended to keep as is, and the local resources are supposed to be wrapped in media queries.

HTML scripts and styles

<link rel="icon" href="./public/favicon.ico" />

<script src="src/index.ts"></script>

In the case above, href in link tags and source in script tags should be treated as dependencies and these references to the asset location should be replaced.

Experimental: Import Assertions

Proposal

import json from "./foo.json" assert { type: "json" };
import("foo.json", { assert: { type: "json" } });

Import Assertions will change the module type to asserted type.

Detailed design

trait Dependency

trait Dependency: Hash + Eq + PartialEq {
  const KIND: &'static str;
  fn specifier(&self) -> String;
  fn resolve_kind(&self) -> ResolveKind;
  fn assertion_type() -> Option<AssertionType> {
     None
  };
}


enum AssertionType {
   JSON
}

Basically, a Dependency contains these stuff:

1. const KIND: KIND is an identifier for debugging
2. fn specifier: a specifier targeting the importing module, queries are included.
3. fn resolve_kind: the resolve kind indicates the type of the reference, such as dynamicImport / Import / AtImport, etc.
4. fn assertion_type: the assertion type, which returns None by default. If an assertion is set, a corresponding AssertionType is supposed to be returned.

Since a dependency implements trait Hash, We have to provide an ID generator to create a normalized and unified dependency id.

Dependency collecting and rendering

Asset handling

This part is for those modules that do not have the corresponding runtime and help those to point assets to the generated location, like CSS. JavaScript(with custom runtime) is handled by the commonjs runtime, so this feature is not a must for them.

Rspack collects dependency in each module implementation. Each module is supposed to implement fn dependencies with a boxed struct that implements trait Dependency returned.

trait Module {
  ...
  fn dependencies(&mut self) -> Vec<Box<dyn Dependency>>
}

In the meantime, you can utilize this function(with &mut self), to create a hash replacement for each module, such as in the CSS module.

// POC and this does not represent the final implementation.
struct CSSUrlDependency {}

impl Dependency for CSSUrlDependency {
   ...
}

struct CSSModule {}

impl Module for CSSModule {
  fn dependencies(&mut self) -> Vec<Box<dyn Dependency>> {
     // Parse css dependencies
     ...
     // Create dependency based on the parsing result
     let  url_dep = CSSUrlDependency { .. };
     let id = url_dep.id();

     // IMPORTANT: 
     // You need to replace the `specifier` with `id` generated from each dependency in each CSS URL expression so that rspack can utilize them to replace the `id` with the canonicalized URL later.
     // Also to notice that the `id` is necessary as it is different from the pure replacement based on `specifier` since the id takes something more than `specifier` alone into account and may point to different generated assets. For more information, you may refer to the trait `Dependency` section above.
     // This replacement function should directly affect the render function in each module, but `rspack_core` does not need to know the real implementation.
     call_some_replacement_fn_for_css_url();
     // Finally, we return the parsed dependency list
     vec![Box::new(url_dep)]
  }
}

The returned dependencies will be used to create ModuleGraphModule in the module graph of rspack.
After the render_manifest, the URLs in CSS Modules are replaced with hashes and we collect all the code generation results as assets, so we can directly replace the hash with the real generated locations of the assets.

Imports that affect the module rendering

In those files like CSS, the at-import statement might affect how an importee is included in the bundle. Let's grab the example above.

/* landscape.css */
.landscape {
   background-color: #fff;
}

/* index.css */
@import url('landscape.css') screen and (orientation: landscape);

The code generation result will be as follows:

@media screen and (orientation: landscape) {
  .landscape {
    background-color: #fff;
  }
}

The core idea to implement this in Rspack is to give the dependency in each chunk to the corresponding module render function(you may find this on trait Module) just like this.

// POC, and this does not represent the final implementation.
impl Module for CSSModule {
  fn render(
    &mut self,
    requested_source_type: SourceType,
    module: &rspack_core::ModuleGraphModule,
    compilation: &rspack_core::Compilation,
  ) {
      let module_graph = &compilation.module_graph;
      let chunk_graph = &compilation.chunk_graph;

      // Here we have the access to the module graph and chunk graph on each compilation and grab the incoming dependency to this module.
     // For the example above, the incoming dependencies of `landscape.css` is `vec!["index.css"]`
     let incoming_dependencies_in_chunk_graph = get_incoming_dependency();
     // We can filter the dependencies with `ResolveType::AtImport`, and pass the module render result to the dependency render and return it.

    let module_render_result = SWC_CSS_COMPILER.codegen(&self.ast);
    
    // here we can downcast the dependency and wrap the module with media queries and so on.
  }
}

In the CSS world, at-import with the same specifier could be called with different media queries, which means we have to iterate through the incoming dependency list, and if which is found, we need to do the fallback just as follows:

Detail about CSS support is out of range of this RFC.

In conclusion, we have to support get_incoming_dependency_by_module_uri for the module graph and let the importee handle the dependency rendering logic.

Overriding module type based on import assertions

Since each dependency contains assertion_type, and a new module factory is created with a dependency, so we can directly set the context containing module_type from assertion_type.

Module Identifier

A module identifier in rspack is used as a key to determine whether a module should be reanalyzed or not. Currently, we use a visited_module_identity: HashSet<String> with module URI to avoid duplicated analysis. However, the identifiers used to avoid duplicated behavior in the module creation step and the module render step should be different.

Firstly, I would like to use the import assertion example to describe this, and use module URI as the key:

import unknownJson from "./some-file" assert { type: "json" }
import unknownJs from "./some-file"

In this case, as the URI(specifier) are the same, so rspack will not create a new module for the second line, however, we need to regard the module type of the secondly imported module as javascript. Using the URI as the key stops us from creating a new module. It's a bummer.

If we use dependency id as the key, and since dependency id is generated with assertions taken into account, rspack will generate two modules. Great!

Here's another example. Again, let me use the CSS example to describe this, and at this time, we use dependency id as the key:

In the example above, the CSS file landscape.css is included in multiple files: index.css and foo.css, and in the world of CSS, an at-import with or without the media query does not change the module data like type, and the id of a dependency is not generated based on media query by default. So again, the problem is solved! If later the CSS specification add a new specification, we can still define our custom fn id for a CSS dependency.

In conclusion of the example above, we have to change the key to the id of each dependency.

But what about the module id? Using the dependency id as the module id is ok, but it's not enough. Again, with the CSS example above, we need to find the incoming dependency to a certain specifier not the key used in the HashSet.

Alternatives

You may refer to the prior art section.

Drawbacks

When I first started to design the dependency abstraction, I would like to provide an render method for each dependency. For example, CSS need to wrap media queries around the module render result. But each dependency's rendering logic is different and we don't know the required information for each module type. Webpack's dependency template is only used to patch the asset location. The workaround is to use the downcast for each dependency, and it's ok to do that.

Prior art

How Webpack works?

Glossary

• WebpackAST: a loader can return a WebpackAST, which is forced to be a JavaScript AST, other ASTs like CSS AST is not supported, and webpack can decide whether it should reuse AST.
• Dependency: A dependency is an object that includes the importer and importee and its own metadata like range. It is also worth to notice that dependency included in different module types are different, for example: CSSUrlDependency / CSSImportDependency / HarmonyImportDependency.
• Template: Webpack uses templates to wrap code into runtime templates and transform dependencies based on the runtime requirements, e.g HarmonyImports would be replaced to webpack requires (see: getImportStatement in HarmonyImportDependency)

Example

Here's a snippet of CSS:

@import url("https://cdn/file.css");

body {
  background-image: url("./images/file.png");
}

which generates two dependencies as follows:

CSSImportDependency {
  media: "url",
  range: [.., ..] // which represents the starting and ending point of `@import url(...)`,
  request: "https://cdn/file.css",
  ... // other fields are omitted
}

CSSUrlDependency {
  request: "./images/file.png",
  range: [.., ..] // which represents the starting and ending point of `url(...)`
}

Take a look at the CSSUrlDependency, the URL of which might be replaced with an updated location point to the assetModuleFilename or inline-base64 determined by options. Thus, webpack provides an abstraction on the replacement, which is called a dependency template:

// This is an easy implementation to help people understand what's happening under the hood.
CSSUrlDependency.Template = function (source, range, replacement) {
  var start = range[0]
  var end = range[1]
  return // replace the source from start to end with replacement
}

Architecture conflicts between webpack and rspack

Before going deep into the actual conflict, I would like to dig more into how webpack works from the perspective of ASTs and templates. We assume the module type is javascript/auto for the example below.

Firstly, webpack will read all resources as a UTF-8 string by default. Once a loader is encountered, you can transform the code into an
ESTree, then you may invoke the loader callback with the code or webpackAST(passed as a property for the fourth parameter). If the AST is returned, the parser will directly analyze this AST, or webpack will parse the code return invoked with the callback and continue the analysis procedure.

Secondly, after all the analysis is done, webpack applies templates that were registered with code presented as module.originalSource() and patches the code, then emit the assets.

In conclusion, webpack basically handles string values everywhere. Like how webpack does, most bundlers collect dependency after the transform. However, Rspack handles AST directly and generates code based on ASTs, which basically means if we collect dependencies with range we cannot easily replace the string created with range by the dependency template since the code generation results may be different. Another approach for this is to do the code generation after each transformation, however this approach is not decent as code generation logic should not be included in the build step.

Conclusion

Webpack bundle system is built on string patches. The abstraction of range is not quite suitable with the rspack architecture, and an AST-friendly approach is necessary.

How parcel works?

Rspack's transformer architecture is more likely to parcel's. In parcel's transformer architecture, a transformer may add dependencies manually just like this:

import {Transformer} from '@parcel/plugin';

export default new Transformer({
  async transform({asset}) {
    let code = await asset.getCode();
    let result = code.replace(/import "(.*?)"/g, (m, dep) => {
      // Replace the original specifier with a dependency id 
      // as a placeholder. This will be replaced later with 
      // the final bundle URL.
      let depId = asset.addURLDependency(dep);
      return `import "${depId}"`;
    });

    asset.setCode(result);
    return [asset];
  }
});

addURLDependency creates a dependency bounded to the asset internally and returns the placeholder. The hash is calculated based-on the follows. You may refer to this to check out the real implementation.

let id =
    opts.id ||
    hashString(
      (opts.sourceAssetId ?? '') +
        opts.specifier +
        opts.env.id +
        (opts.target ? JSON.stringify(opts.target) : '') +
        (opts.pipeline ?? '') +
        opts.specifierType +
        (opts.bundleBehavior ?? '') +
        (opts.priority ?? 'sync'),
    );

After the code generation, parcel replaces the placeholder with real generated paths, link

Unresolved questions

1. In current RFC, finding incoming dependencies is placed in a module render. If we filter them outside the module render, then it will affect the JavaScript module renders, otherwise it's not quite decent. Maybe a better solution in the future? But it's ok for now.
2. Can we place assertion in the implementation rather than placing it in the trait? How is it going to interfere with the module type modification?

[byted-meow]

As we are about to support HTML entry, will inline stylesheets and scripts be considered as dependencies? And how to represent them?

<style>
  /* some styles */
  @import url('./src/some-style.css');
</style>
<script>
  // some script
</script>
<script type="module">
  // some module script
  import * as script from './src/some-script.js'
</script>
<script type="x-shader/x-fragment">
  // some shader
</script>

[h-a-n-a]

It need support resolve path and resource query.

Definitely! We should align the naming to webpack, like resource / resourceQuery / resourcePath. I think this is a standalone part that can be discussed in another RFC.

[h-a-n-a]

Maybe we should consider supporting html dependency parsing, it can be useful in some situations. Like the following examples

// common case icon loading
<link rel="icon" href="./public/favicon.ico" />

// Inject some code directly into the html that you want to execute them first
<script src="./sdk/render_sdk.js?inline"></script>

In these cases, we need to handle them as we would handle resources

I agree with you, and the HTML module should implement the fn dependencies which returns all dependencies, and we use these dependencies to create module creation jobs.

[liuliudada-w]

Great! If so, we need add some examples to Affected examples such as HTMLSCRIPTSRC、HTMLLINKHREF and more

[h-a-n-a]

Great! If so, we need add some examples to Affected examples such as HTMLSCRIPTSRC、HTMLLINKHREF and more

Sure! Added this.

[ahabhgk]

If we want to treat HTML as first-class citizen, any local resources(styles / scripts / assets / or even HTMLs included in this page) should be create as a separate dependency. The inlined scripts and styles are supposed to be treated like standalone files and inserted back into the HTML.

Agree with this.

For more information about the detailed implementation as I would expect, you can refer to parcel's HTML

About the detailed implementation, the parcel's HTML uses one AssetGroup which includes HTML asset and inlined JS asset to represent the inline relationship, and uses a separate AssetGroup (one for HTML, another one for no-inlined JS) to represent the no-inline relationship as a usual dependency.

No inline:

<html>
  <head>
    <script src="./index.js"></script>
  </head>
</html>

Parcel AssetGraph:

Inline:

<html>
  <head>
    <script>console.log('hahah')</script>
  </head>
</html>

Parcel AssetGraph:

How parcel organizes the AssetGraph might be over complex for us, Could we use Dependency::Kind or impl InlineDependency for Dependency to represent the inline relationship rather than how parcel does? Just want to make it more clear.

[liuliudada-w]

Dependency abstraction is a good solution to the problem that the same module behaves differently in different ways of reference. What I'm more worried about is that Dependency abstraction make module relationship changing from module --> module to module --> dependency --> module. It will cause many changes in the operation of the module graph.

[h-a-n-a]

It's almost the same implementation as before. In this RFC, We replace the dependency list with a list of Box<dyn Dependency> to separate different dependency types and the relationship in the module graph is almost the same. The relationship in the module graph is like this: ModuleGraphModule -> ModuleGraphModule. You can get all dependencies in a module graph module and utilize this to find its dependencies and the related modules.

[hyf0]

Do we need fn id(&self) -> String to identify whether a struct is the same? Is the Hash trait enough?

[h-a-n-a]

After an offline discussion with @iheyunfei , fn id will be removed, and use a separate id generator to unify the result.