Metro: Versioning and resolution of "builtin modules"

[motiz88]

I'd like to share a problem that has come up in a couple of contexts, and my initial thoughts about a possible solution.

At a high level, the problem is that the transformer, resolver, package manager (e.g. npm / yarn) and runtime environment (e.g. React Native) are coupled together by a set of assumptions about versioning and resolution that don't always hold.

Motivating example

When we run the Babel plugin that transpiles class syntax on the following file:

// MyProject/src/components/Foo.js (source)

class X {}

it will produce something like the following code:

// MyProject/src/components/Foo.js (transformed)

const _createClass = require("@babel/runtime/helpers/createClass");
const _classCallCheck = require("@babel/runtime/helpers/classCallCheck");

var X = _createClass(function X() {
  _classCallCheck(this, X);
});

This makes intuitive sense. But those require("@babel/runtime/...") calls are evaluated in the context of MyProject/src/components/Foo.js. Let's look at that fact more closely.

Things that can go wrong

Under the default resolution algorithm, what will typically happen is that we'll traverse the directory tree upwards from MyProject/src/components, look for directories named node_modules/@babel/runtime along the way, and eventually find a match. But this process has a number of failure modes baked into it:

1. We could resolve to an older or newer version of @babel/runtime than we intended - e.g. one that doesn't have the createClass and classCallCheck helpers, or has them under different paths (thus causing build errors), or has them with different semantics (thus causing runtime errors, or worse - silently inconsistent behaviour).
2. We could resolve to multiple copies of @babel/runtime in multiple parts of our project and inadvertently bloat the bundle.
3. We could fail to find @babel/runtime altogether and thus fail to build.
4. We could resolve @babel/runtime via a symlink to a location that Metro isn't watching for the current project, and fail to build that way.
5. There could also be a custom resolver that changes the resolution of @babel/runtime in some arbitrary way, either globally or based on the provided context.
6. Finally, the user might want to customise the version of @babel/runtime they use in order to produce optimal bundles; but if they accidentally set the wrong version in enableBabelRuntime compared to what they actually have installed, things can break (see 1).

In other words, what we have here is a leaky abstraction; Metro should be able to guarantee correctness and Just Work™️, but in practice this responsibility falls to the user. At minimum, we kind of implicitly expect users not to do any of the things that don't work.

Note that this problem is far from unique to Babel helpers. Similar failure modes arise whenever the transformer generates code that assumes anything about how modules are resolved - something that the transformer fundamentally, architecturally, can't predict. Other examples include:

• Coordinating the versions of the Fast Refresh Babel plugin (shipped as part of Metro) with the Fast Refresh runtime (shipped as part of React Native)
• Matching the version of metro-runtime (in particular the require() implementation) to that assumed by the Metro transformer and serializer.

Partial solution: Built-in modules

Instead of relying on the user's project to be set up a certain way, let's think of certain packages as being built into Metro. After all, Metro's transformer ships with specific, known, versions of the compile time bits of Babel, Fast Refresh, etc; can we make it so this fully determines the versions of their runtime counterparts?

My idea here is inspired by Node.js's node: imports. We can expose a fixed set of "built-in" packages under metro: URIs, e.g.:

• metro:metro-runtime
• metro:@babel/runtime
• metro:react-refresh.

Semantically, importing a built-in package would bypass node_modules resolution and instead always resolve to a version managed and provided directly by Metro. Using this, we can teach the Babel transformer to emit require('metro:@babel/runtime/...') calls, change React Native to require Fast Refresh as require('metro:react-refresh/runtime'), etc.

Implementation details

The physical location of each package would be determined by using require.resolve from within one of the core Metro packages. Metro's package.json would declare the dependencies on the builtins so users don't have to (and in fact can't interfere with the builtins, even if they do install their own conflicting versions).

To make this work reliably, we might need to transparently add the physicals locations of these packages (and any of their own dependencies) to the active project's file map, so they can be referenced even if they happen to be outside of the current projectRoot or its configured watchFolders.

SIDE NOTE: One wrinkle is that merely adding something to watchFolders can - in edge cases - pollute the Haste map with Haste package names and thus affect non-metro: resolutions. We can work to untangle this now that we ship our own metro-file-map instead of depending on jest-haste-map.

TBD whether we can/should invoke the custom resolver on metro: imports.

More general solution: Transformer-provided modules

Including a closed set of built-in modules with Metro would work for the specific use cases I listed above - but does it scale to custom transformers, custom Babel plugins, etc? Ultimately any sufficiently complex transformer can run into this problem.

Maybe we can extend the Metro transformer API to let custom transformers register their own custom "built-ins" with the same general contract as metro: imports: the resolution is not up to the user's project but up to the toolchain that is building the user's project.

Ideally, this API would be expressive enough that we can do things like:

• Implement the "standard" metro: packages on top of it
• Only include the Fast Refresh runtime as a built-in when metro-react-native-babel-transformer is in use.