Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

import in referenced typescript project fail to load #96

Open
1 task done
steabert opened this issue Sep 2, 2022 · 20 comments · May be fixed by esbuild-kit/esm-loader#57 or esbuild-kit/cjs-loader#32
Open
1 task done
Labels
bug Something isn't working pending triage tsconfig

Comments

@steabert
Copy link

steabert commented Sep 2, 2022

Bug description

When working with composite projects, our tests failed because it seems that tsx is not taking the reference project's baseUrl or path into consideration.

Reproduction

Test project:

https://github.com/steabert/esbuild-kit-tsx-composite.git

To reproduce the issue:

yarn install
yarn fail

Running yarn bundle and yarn types shows the result with e.g. esbuild or tsc.

It seems this does not work with ts-node either, but I don't know how they handle composite typescript projects.

Environment

npx envinfo --system --npmPackages tsx --binaries
System:
  OS: Linux 5.4 Ubuntu 20.04.4 LTS (Focal Fossa)
  CPU: (8) x64 Intel(R) Core(TM) i5-8250U CPU @ 1.60GHz
  Memory: 23.58 GB / 31.21 GB
  Container: Yes
  Shell: 5.0.17 - /bin/bash
Binaries:
  Node: 16.14.2 - ~/.node/bin/node
  Yarn: 3.2.3 - ~/.npm-global/bin/yarn
  npm: 8.5.0 - ~/.node/bin/npm
npmPackages:
  tsx: 3.9.0 => 3.9.0

Can you contribute a fix?

  • I’m interested in opening a pull request for this issue.
@steabert steabert added bug Something isn't working pending triage labels Sep 2, 2022
@steabert
Copy link
Author

steabert commented Sep 2, 2022

Also, looks similar to #76 maybe?

@IlyaSemenov
Copy link
Contributor

Yes, this is the same problem as reported in #76. unlike esbuild, tsx ignores tsconfig.json in referenced packages.

@privatenumber
Copy link
Owner

How does esbuild handle custom tsconfig.json paths via the tsconfig CLI flag for multiple projects?

e.g. If project A uses tsconfig.a.json and its dependency project B uses tsconfig.b.json, how is it configured?

@steabert
Copy link
Author

steabert commented Sep 30, 2022

I don't need to specify any configuration, esbuild seems to find the correct tsconfig associated with an imported module. However, I think it only does that when bundling. I forgot where I read that, I think it was in the comments on one of the esbuild issues, I'll try to find it again.

Edit: here was the comment that I found saying what the behaviour is, but nothing in detail how it's resolved.

@steabert
Copy link
Author

My own (naive?) way would be to walk up the directory tree of an imported file if there is no associated tsconfig, and if a new tsconfig.json file is found, check its include files and cache it for later use. That way, each included file would have an associated tsconfig file.

@privatenumber
Copy link
Owner

My question is more about API design than your specific use-case.

Understanding how esbuild's API handles custom tsconfig.json paths across multiple projects would help.

@steabert
Copy link
Author

steabert commented Oct 3, 2022

The esbuild author replied in my issue, but that seems to be also about how esbuild itself finds tsconfig files. That doesn't seem to be the info you're after, I'm afraid I haven't dealt with the esbuild API at all.

@privatenumber
Copy link
Owner

To be fair, you asked him a different question from the one I asked you.

I don't have time for this now, but if you're interested in helping, you can investigate by setting up a project with the structure I described here and trying to make it work with esbuild. Maybe @IlyaSemenov can help.

Based on what Evan said, I'm skeptical if esbuild is also following TS behavior correctly (not that tsx currently is either). For example, does it even allow for custom tsconfig.json paths? And does it check against tsconfig.json#files/includes/excludes?

@mtqperson
Copy link

mtqperson commented Oct 5, 2022

We are also running into this problem. I might be able to help but I'm not sure what you mean by

you can investigate by setting up a project with the structure [...] and trying to make it work with esbuild.

How does esbuild handle custom tsconfig.json paths via the tsconfig CLI flag for multiple projects?
e.g. If project A uses tsconfig.a.json and its dependency project B uses tsconfig.b.json, how is it configured?

Like @steabert described, with esbuild it already works because esbuild detects tsconfig.b.json by itself. Do you want to know whether it still works if we additionally specify tsconfigs manually, even though that's not required?

@privatenumber
Copy link
Owner

I'm skeptical that esbuild would successfully build. My understanding is that it only detects tsconfig.json, and not tsconfig.b.json.

To prove that this works, please create a repository containing a monorepo with two tsconfigs under custom names and add a build script to demonstrate that esbuild can successfully build it. If you'd like, you can reuse https://github.com/steabert/esbuild-kit-tsx-composite but rename the tsconfig.json of the dependency and test if it builds.

And if that doesn't build, please find out what configuration esbuild offers to specify a custom tsconfig.json name.

@IlyaSemenov
Copy link
Contributor

This issue is a blocker for any monorepo with aliases, which is (arguably) more popular setup than multiple custom named tsconfig's. Also, the current behaviour is not in sync with how esbuild works itself. In my reproduction repo tsx fails to run yet esbuild compiles it just fine:

❯ p esbuild test.ts --bundle > test.js

❯ cat test.js
"use strict";
(() => {
  // ../b/lib.ts
  console.log("imported lib");
})();

❯ node test.js
imported lib

I would like to try to come up with a patch as I'm not happy with my current setup (nodemon + esbuild-runner) and looking for alternatives. @privatenumber do you think you can point me in the right direction, if you foresee any?

@privatenumber
Copy link
Owner

We should figure out how custom tsconfig paths are handled in nested projects: #96 (comment)

@IlyaSemenov
Copy link
Contributor

I am not following why the custom configs matter so much? It would suffice if nested projects worked at least with default configs. Supporting custom configs in nested projects could be the Phase 2 implemented by whoever cares about that use case.

Chances are esbuild doesn't support custom configs in nested projects at all, then what? That doesn't prevent it from working with default configs; and neither should it prevent tsx from supporting the same case.

@privatenumber
Copy link
Owner

I am not following why the custom configs matter so much? It would suffice if nested projects worked at least with default configs.

Why it matters: #76 (comment)

tsx has a lot of users and handles many use-cases. Custom tsconfig files are very common so I think the more appropriate question you should be answering is "why should we dismiss those use-cases without even doing an hour of research?".

Supporting custom configs in nested projects could be the Phase 2 implemented by whoever cares about that use case.

FYI, I don't benefit from this use-case but I'm happy to implement it in the future. With regards to your attitude, I'm wondering why you haven't implemented this yourself? As with anything open-source, you should also feel free to implement it yourself and use your own fork.

Chances are esbuild doesn't support custom configs in nested projects at all, then what? That doesn't prevent it from working with default configs; and neither should it prevent tsx from supporting the same case.

That's fine too. But no one has provided a definitive answer with research.

When I decide to take this on, I would like to think of the best solution possible and would investigate different use-cases and solutions. Since ya'll are asking how you can help, I provided one of the questions I would research.


I'm not sure how this thread will be productive if my answers to your "how can I help?" is met with "why is that important?".

If no one is going to provide insight into potential problems and how other projects handle it, it might be best to lock this thread until I'm ready to tackle it and do the research myself.

@IlyaSemenov
Copy link
Contributor

Thank you for the response. I apologize if I sounded rude. I know how open-source works, I authored and support a number of npm and python modules myself, and I am happy to provide users the right direction if they ask (or point them where they're wrong).

My point here was that making fully broken system half-broken would still be a good thing. I gathered that the current implementation blocker was that it wasn't clear how to configure esbuild to use different tsconfig's in different sub-projects. In my thinking that should not be a blocker even if that's not possible.

I did make the initial research back then when you asked the first time and it immediately appeared that the public esbuild API only allowed to customize the root/entrypoint tsconfig: https://esbuild.github.io/api/#tsconfig
I didn't share that finding as it appeared too obvious for me.

In light of that, I was merely asking if you could provide your initial high level thoughts of where the source could be extended to load paths from nested tsconfigs (considering the limitation of not being able to use custom-name tsconfig), outline the frames of the possible required architectural changes. I surely understand it's not always possible to answer that kind of questions, but sometimes it is.

@jgoux
Copy link

jgoux commented Jan 31, 2023

Hello everyone 👋,

I also hit this issue. There is no way to specify custom tsconfig.<whatever>.json config files for external packages bundled with esbuild. It only looks for the regular tsconfig.json.

As far as I understand, just supporting the regular tsconfig.json files would solve my issue (I'm using a pnpm monorepo, I have the regular tsconfig.json for vscode + dev workflow, and a tsconfig.build.json for build). For that, we should not pass an explicit --tsconfig option when calling esbuild, as it will use this single tsconfig file and ignore all the others.

So paths defined in tsconfig.json can work but in a limited way (every tsconfig files need to be named tsconfig.json, including the root one).

Do you think we could add an option to enable this behavior? Maybe --resolve-tsconfigs or something like this?

I'm happy to make a PR if you can point me to the relevant files! 😄

@jgoux
Copy link

jgoux commented Feb 2, 2023

If people are stuck on this, I had success with vite-node combined with vite-tsconfig-paths.

It also plays nice with vitest as it's reusing the same pipeline. 👍

@jgoux
Copy link

jgoux commented Feb 18, 2023

I'm very sorry for this triple post, but I played some more with my TypeScript monorepo and I think I gained interesting insights for this issue.

Big picture

tldr; TypeScript project references are the way to hint about which tsconfig.json file to apply to parts of the dependency chain

First, the "one true way" of connecting TypeScript projects together is project references. This is the only mechanism that enables TypeScript to split the dependency chain between multiple TypeScript projects and use each project's tsconfig.json.

Custom tsconfig.json files are handled by this mechanism as well:

{
    "compilerOptions": {
        // The usual
    },
    "references": [
        { "path": "../api/tsconfig.build.json" },
        { "path": "../sdk/tsconfig.whatever.json" },
    ]
}

Without references, if you require a TypeScript file from an external package (through either monorepo workspaces or paths alias, I personally recommend avoiding paths alias as you break your package isolation), only the tsconfig.json file of the root package will be used, this is why external packages paths alias break, because the root package is not aware of their internals (and it should not be 😄).

How to have this implemented in tsx

Project references were discussed in esbuild repository, and the conclusion was that it's beyond the scope of esbuild, but could be implemented as a plugin!

A plugin exists: esbuild-plugin-ts-references but it's not working on paths alias, it's only rewriting the import statements to link directly to the references. But it shows that it's possible with esbuild API to walk through each package's tsconfig.json by following references and build some kind of map.

So the solution would be to write an esbuild plugin similar to esbuild-plugin-ts-references that can rewrite imports based on the file's local tsconfig.json gathered through references analysis. The main blocker is that tsx is using esbuild's Transform API and not the Build API, so it can't consume a plugin.

@privatenumber How hard of a requirement is it to use the Transform API vs the Build API? Reading esbuild doc it seems like Transform is just a small subset of Build?

Another solution would be to have tsx do the references analysis work outside of esbuild to build the map and then apply the rewrites somehow with the Transform API, but I'm not familiar with it enough to know if it's possible. 😅

edit: It seems like if we can construct the map <tsconfigPath, filePaths>, we could be able to pass the right tsconfig.json content here: https://github.com/esbuild-kit/cjs-loader/blob/94a69f9e6d41225cb8a132938a3b9b077c15f966/src/index.ts#L74

edit 2: I gave my "projects map" idea a try, I think I'm pretty close to making it works but I have a nasty recursive bug somewhere that I can't figure out. 🤔

But I'm able to :

Please note that it's not "performance" oriented, right now I'm just trying to understand the code and make it works. 😅

Here are the forks :

And I test against this project:

edit 3: My recursion bug is fixed and the code for https://github.com/jgoux/esm-loader/tree/feat-project-references is fully functional! 🎊

@jgoux
Copy link

jgoux commented Feb 20, 2023

I opened PRs to esm-loader and cjs-loader to support project references which should solve this issue.

@steabert @IlyaSemenov If you can give it a try it would be great! 👍

@pokey
Copy link

pokey commented Mar 10, 2023

I was able to get this working using a custom exports field on all my monorepo package.json files, and then passing --conditions to tsx:

  "exports": {
    ".": {
      "myOrganizationName:bundler": "./src/index.ts",
      "default": "./out/index.js"
    }
  }

Then

tsx --conditions=myOrganizationName:bundler ...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working pending triage tsconfig
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants