Improve WASI support

[hyf0]

This is a followed up of #832.

With the hard work of @Brooooooklyn, we now are ok to compile to wasi successfully and run wasi rolldown successfully in some simple cases.

However, there are still a lot of hard and wired problems to be solved.

As #894 showed

• In unbuntu
  • The wasi rolldown could compile the basic example successfully
  • Can't pass tests due to being hang while running
• In windows/macos
  • Unsupported Operation error was throwed in rust even in the basic example case

Other problems

rolldown/.github/workflows/reusable-wasi-test.yml

Lines 43 to 47 in 7b64957

	# Node.js wasi runtime has bug with `std::fs::read_link`
	run: |
	touch .npmrc
	echo "node-linker=hoisted" >> .npmrc
	pnpm install

rolldown/crates/rolldown_binding/src/bundler.rs

Lines 35 to 37 in 7b64957

	#[cfg(target_family = "wasm")]
	// if we don't perform this warmup, the following call to `std::fs` will stuck
	if let Ok(_) = std::fs::metadata(std::env::current_dir()?) {};

Update: oh no! When set RUST_LOG=trace, pnpm --filter=@example/basic build success, fails if unset RUST_LOG, so strange! #898 (comment)

4. Running a playground on the browser is blocked by: tokio: enable rt-multi-thread for wasm32-wasi-preview1-threads target tokio-rs/tokio#6510. Because the fs polyfill crosses the main thread/ web worker is using Atomic.wait, any std::fs operations within the async fn will block tokio current_thread runtime forever.

[toyobayashi]

@Brooooooklyn @hyf0

I can build successfully on Windows by manually make following changes:

examples/basic-vue/rolldown.config.js

  import { defineConfig } from 'rolldown'
  
  export default defineConfig({
    input: './index.js',
+   // set posix path for WASI, not native platform path
+   // map to C:\\Users\\toyobayashi\\Projects\\rolldown\\examples\\basic-vue
+   cwd: '/Users/toyobayashi/Projects/rolldown/examples/basic-vue',
    resolve: {
      // This needs to be explicitly set for now because oxc resolver doesn't
      // assume default exports conditions. Rolldown will ship with a default that
      // aligns with Vite in the future.
      conditionNames: ['import'],
    },
  })

packages/rolldown/dist/shared/rolldown-binding.wasi.cjs:

  const __wasi = new __nodeWASI({
    version: 'preview1',
    env: process.env,
    preopens: {
-     '/': '/'
+     '/': 'C:\\'
    }
  })

packages/rolldown/dist/shared/wasi-worker.mjs:

const handler = new MessageHandler({
  onLoad({ wasmModule, wasmMemory }) {
    const wasi = new WASI({
      version: 'preview1',
      env: process.env,
      preopens: {
-        '/': '/'
+        '/': 'C:\\',
      },
    });

[hyf0]

@toyobayashi Thanks. This should be a windows-specific problem. I guess we should give a sensible default value for windows.

[toyobayashi]

Just now I also built basic example successfully on macOS without modify anything (b5b06bf). Just ensure the vue in node_modules is a real directory instead of a symlink :)

Update: oh no! When set RUST_LOG=trace, pnpm --filter=@example/basic build success, fails if unset RUST_LOG, so strange!

[Brooooooklyn]

Update: oh no! When set RUST_LOG=trace, pnpm --filter=@example/basic build success, fails if unset RUST_LOG, so strange!

We observed this issue too, and this line is also strange:

rolldown/crates/rolldown_binding/src/bundler.rs

Lines 35 to 37 in 7b64957

	#[cfg(target_family = "wasm")]
	// if we don't perform this warmup, the following call to `std::fs` will stuck
	if let Ok(_) = std::fs::metadata(std::env::current_dir()?) {};

[Brooooooklyn]

Can confirm there are some compiler bugs in Rust. Switch to nightly-2024-04-18 can make the example work without RUST_LOG=trace

[hyf0]

Sorry to say that #975 breaks building basic-example in unbuntu. I can't figure out why with the changed code should not affect wasm.

[mdarrik]

So I was able to repair the symlink stuff for MacOS by using the same canonicalize logic that oxc-resolver uses in their fs implementation. It seems that dunce uses std::fs::canonicalize on non windows targets, which isn't available on WASI yet.

// modified from oxc-project/oxc-resolver/src/file_system.rs

    #[cfg(not(target_family = "wasm"))]
    {
      dunce::canonicalize(path)
    }
    #[cfg(target_family = "wasm")]
    {
      let meta = std::fs::symlink_metadata(path)?;

      if meta.file_type().is_symlink() {
        let link = std::fs::read_link(path)?;

        let mut path_buf = path.to_path_buf();
        path_buf.pop();
        for segment in link.iter() {
          match segment.to_str() {
            Some("..") => {
              path_buf.pop();
            }
            Some(".") | None => {}
            Some(seg) => {
              // Need to trim the extra \0 introduces by rust std rust-lang/rust#123727
              path_buf.push(seg.trim_end_matches('\0'));
            }
          }
        }
        Ok(path_buf)
      } else {
        Ok(path.to_path_buf())
      }
    }

No luck yet on tracking down the need for RD_LOG to get WASI builds to run.

I did notice I get a different panic if I force WASM to init the tracing-subscriber by skipping the short-circuit return None in the tracing subscriber.

• Without any changes so WASM doesn't init a tracing_subscriber, I get RangeError: Invalid atomic access index
• With changes so WASM always inits a changing subscriber, I get Error [RuntimeError]: null function or function signature mismatch.

[nabby27]

I like the idea of ​​rolldown and wanted to contribute to this issue by creating a $200 bounty on Opire. It's my way of being able to contribute my grain of sand since I don't have the knowledge to be able to help in any other way.

[yyx990803]

@nabby27 thanks for the interest and support - for now we are focusing on the feature side since this will require some dedicated effort to be resolved. We will revisit this once we are close to feature complete.

[nabby27]

@yyx990803 Sure, don't worry, I fully understand that the main team is focused on other more important tasks, that's why I created this bounty in case any other developer wants to join the community and solve this issue.