Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hitting TS Limits: "Max Call Stack Size Exceeded" #58439

Closed
texttechne opened this issue May 5, 2024 · 10 comments
Closed

Hitting TS Limits: "Max Call Stack Size Exceeded" #58439

texttechne opened this issue May 5, 2024 · 10 comments
Labels
Needs More Info The issue still hasn't been fully clarified

Comments

@texttechne
Copy link

🔎 Search Terms

"max call stack size"

Related Issues:

🕗 Version & Regression Information

  • This is a crash

⏯ Playground Link

https://github.com/texttechne/max-call-stack-size-exceeded

💻 Code

Unfortunately, it seems that this is a matter of complexity, so there's no "minimal" example.
However, I created a repo to reproduce this issue.

This bug occurs in connection with a generator (odata2ts) which creates an OData client out of metadata. As the metadata of the OData service represents an entity relationship model, there's also some form of recursion involved when entities have a bidirectional relationship to each other. However, this seems fine when the OData service in question is not that large.

The offending code examples:

export class AppointmentService<
  ClientType extends ODataHttpClient
> extends EntityTypeServiceV4<
  ClientType,
  Appointment,
  EditableAppointment,
  QAppointment
> {
  private _AppointmentType?: AppointmentTypeService<ClientType>;
  //...
  
  public AppointmentType(): AppointmentTypeService<ClientType> {
    if (!this._AppointmentType) {
      const { client, path } = this.__base;
      this._AppointmentType = new AppointmentTypeService(
        client,
        path,
        "AppointmentType"
      );
    }

    return this._AppointmentType;
  }
  // ...
}

And:

export class AppointmentService<
  ClientType extends ODataHttpClient
> extends EntityTypeServiceV4<
  ClientType,
  Appointment,
  EditableAppointment,
  QAppointment
> {
  // ...
  
  public AppointmentParties(): AppointmentPartyCollectionService<ClientType>;
  public AppointmentParties(
    id: AppointmentPartyId
  ): AppointmentPartyService<ClientType>;
  public AppointmentParties(id?: AppointmentPartyId | undefined) {
    const fieldName = "AppointmentParties";
    const { client, path } = this.__base;
    return typeof id === "undefined" || id === null
      ? new AppointmentPartyCollectionService(client, path, fieldName)
      : new AppointmentPartyService(
        client,
        path,
        new QAppointmentPartyId(fieldName).buildUrl(id)
      );
  }
}

To be clear: The code itself works in smaller examples. However, at a certain point this code seems to hit the limits of TS...

I have an example in the repo where I could leave out the generation of either 1 or 2 and the error won't occur. There are other services where I would need to leave out both to prevent the error from ocurring.

🙁 Actual behavior

TS crashes with "RangeError: Maximum call stack size exceeded"

Unfortunately the stack trace changes from example to example, so please refer to the mentioned repo.

🙂 Expected behavior

It shouldn't crash

Additional information about the issue

To solve the issue for the odata2ts generator, I've decided to use @ts-no-check in every generated file. On the one hand, no consumer would like to type check generated code and it's hopefully a workaround.

However, I've thought that this might serve as a nice repro case for the underlying issue.

@DanielRosenwasser
Copy link
Member

While I think someone can take a look after the weekend, can you post any of the specific call stacks that you saw?

@texttechne
Copy link
Author

Stacktraces

The example from the repo:
RangeError: Maximum call stack size exceeded
    at String.replace (<anonymous>)
    at Object.toFileNameLowerCase [as getCanonicalFileName] (C:\Users\h\project\playground\max-call-stack-exceeded\node_modules\typescript\lib\tsc.js:913:46)
    at getCanonicalFileName (C:\Users\h\project\playground\max-call-stack-exceeded\node_modules\typescript\lib\tsc.js:119641:17)
    at toPath (C:\Users\h\project\playground\max-call-stack-exceeded\node_modules\typescript\lib\tsc.js:5559:10)
    at toPath3 (C:\Users\h\project\playground\max-call-stack-exceeded\node_modules\typescript\lib\tsc.js:118057:12)
    at getResolvedProjectReferenceToRedirect (C:\Users\h\project\playground\max-call-stack-exceeded\node_modules\typescript\lib\tsc.js:119430:78)
    at getRedirectReferenceForResolution (C:\Users\h\project\playground\max-call-stack-exceeded\node_modules\typescript\lib\tsc.js:118012:22)
    at Object.getModeForUsageLocation2 [as getModeForUsageLocation] (C:\Users\h\project\playground\max-call-stack-exceeded\node_modules\typescript\lib\tsc.js:120537:36)
    at resolveExternalModule (C:\Users\h\project\playground\max-call-stack-exceeded\node_modules\typescript\lib\tsc.js:46951:83)
    at resolveExternalModuleNameWorker (C:\Users\h\project\playground\max-call-stack-exceeded\node_modules\typescript\lib\tsc.js:46928:61)
From a previous odata2ts version (contained some bugs that have been resolved now). Also, this is directly from ts-morph trying to output js + dts
RangeError: Maximum call stack size exceeded
    at isTypeReferenceWithGenericArguments (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:63953:49)
    at getRelationKey (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:63992:14)
    at recursiveTypeRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62324:20)
    at isRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:61862:124)
    at isRelatedToWorker2 (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:63623:18)
    at compareSignaturesRelated (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:61254:16)
    at signatureRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:63612:16)
    at signaturesRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:63514:29)
    at structuredTypeRelatedToWorker (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62982:26)
    at structuredTypeRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62439:23)
    at recursiveTypeRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62409:21)
    at isRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:61862:124)
    at isPropertySymbolTypeRelated (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:63178:16)
    at propertyRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:63233:25)
    at propertiesRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:63450:31)
    at structuredTypeRelatedToWorker (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62971:23)
    at structuredTypeRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62439:23)
    at recursiveTypeRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62409:21)
    at isRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:61862:124)
    at eachTypeRelatedToType (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62198:30)
    at unionOrIntersectionRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62040:176)
    at structuredTypeRelatedToWorker (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62597:25)
    at structuredTypeRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62439:23)
    at recursiveTypeRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62409:21)
    at isRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:61862:124)
    at isPropertySymbolTypeRelated (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:63178:16)
    at propertyRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:63233:25)
    at propertiesRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:63450:31)
    at structuredTypeRelatedToWorker (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62971:23)
    at structuredTypeRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62439:23)
    at recursiveTypeRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62409:21)
    at isRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:61862:124)
    at eachTypeRelatedToType (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62198:30)
    at unionOrIntersectionRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62040:176)
    at structuredTypeRelatedToWorker (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62597:25)
    at structuredTypeRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62439:23)
    at recursiveTypeRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62409:21)
    at isRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:61862:124)
    at isPropertySymbolTypeRelated (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:63178:16)
    at propertyRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:63233:25)
    at propertiesRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:63450:31)
    at structuredTypeRelatedToWorker (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62971:23)
    at structuredTypeRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62439:23)
    at recursiveTypeRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62409:21)
    at isRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:61862:124)
    at eachTypeRelatedToType (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62198:30)
    at unionOrIntersectionRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62040:176)
    at structuredTypeRelatedToWorker (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62597:25)
    at structuredTypeRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62439:23)
    at recursiveTypeRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62409:21)
Using a private repo example
RangeError: Maximum call stack size exceeded
    at isRelatedTo (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:59646:25)
    at isPropertySymbolTypeRelated (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:61084:14)
    at propertyRelatedTo (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:61139:23)
    at propertiesRelatedTo (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:61356:29)
    at structuredTypeRelatedToWorker (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:60877:21)
    at structuredTypeRelatedTo (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:60325:21)
    at recursiveTypeRelatedTo (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:60295:19)
    at isRelatedTo (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:59743:122)
    at eachTypeRelatedToType (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:60084:28)
    at unionOrIntersectionRelatedTo (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:59921:174)
Using yet another private repo example
RangeError: Maximum call stack size exceeded
    at isRelatedTo (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:59646:25)
    at isRelatedToWorker2 (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:61529:16)
    at compareSignaturesRelated (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:59128:14)
    at signatureRelatedTo (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:61518:14)
    at signaturesRelatedTo (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:61420:27)
    at structuredTypeRelatedToWorker (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:60888:24)
    at structuredTypeRelatedTo (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:60325:21)
    at recursiveTypeRelatedTo (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:60295:19)
    at isRelatedTo (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:59743:122)
    at isPropertySymbolTypeRelated (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:61084:14)

@fatcerberus
Copy link

All those stacks are different which indicates they are (probably) seperate bugs.

@ahejlsberg
Copy link
Member

I tried to reproduce using the repo at https://github.com/texttechne/max-call-stack-size-exceeded, but the npm run generate command fails with the following:

> max-call-stack-exceeded@1.0.0 generate
> odata2ts

Loaded config file:  odata2ts.config.ts
---------------------------
Starting generation process. Service name "Dynamics"
Didn't find metadata file at:  odata/dynamics.xml
Input source [odata/dynamics.xml] doesn't exist!

@ahejlsberg ahejlsberg added the Needs More Info The issue still hasn't been fully clarified label May 8, 2024
@texttechne
Copy link
Author

Hi @ahejlsberg,

so sorry to have messed up the example like this.

Generation works now, but you don't need to generate the code. I've committed it directly under folder build.
So you can run npm run test-casePacer directly to see the mentioned error. The other two examples do work and are there for comparison.

@ahejlsberg
Copy link
Member

Ok, I can reproduce the issue. This looks to be a case of very deep type dependencies for which the compiler is attempting to compute variance information. For example, from top to bottom, each type below depends on the next type in the list, and it keeps going for >150 levels:

FirmCustomFieldService<ClientType>
CustomFieldTypeService<ClientType>
FirmService<ClientType>
PartyService<ClientType>
DocumentTemplateService<ClientType>
PartyTypeService<ClientType>
LawsuitPartyTypeService<ClientType>
DepositionService<ClientType>
LawsuitPartyService<ClientType>
LawsuitService<ClientType>
AppointmentService<ClientType>
ADRService<ClientType>
ADRTypeService<ClientType>
LitigationService<ClientType>
LawsuitPhaseTypeService<ClientType>
AnswerEnlargementService<ClientType>
LawsuitPhaseDateService<ClientType>
ComplaintService<ClientType>
SubPhaseTypeService<ClientType>
DefendantLitigationAssnService<ClientType>
DefendantLawsuitPartyService<ClientType>
DepositionLitigationAssnService<ClientType>
LawsuitTypeService<ClientType>
TicklerTemplateService<ClientType>
AccountingSyncGLService<ClientType>
RoleTemplateService<ClientType>
CasePacerUserService<ClientType>
CaseMenuItemService<ClientType>
CaseMenuGroupService<ClientType>
CaseMenuTabService<ClientType>
CPDirectCaseProgressService<ClientType>
RecordFileService<ClientType>
RecordFolderService<ClientType>
RecordSubtypeService<ClientType>
IntakeClientLawsuitService<ClientType>
...

Since variance computation is a recursive process, the deep dependencies eventually cause a stack overflow. I was going to suggest working around the problem using explicit variance annotations, but unfortunately the ClientType type parameter in each of these types ends up being bivariant and we lack syntax to express that in an annotation.

I'm not sure how important it is for these types to be generic. If the ClientType type parameter was eliminated, there likely wouldn't be an issue.

@texttechne
Copy link
Author

Thanks so much! Good to know what the actual problem is. I was definitely lost there...

So I've tried out your suggestion to dispense with the ClientType type parameter and tada... it works. It works even with really large metadata files (20 MB). Takes it's time, but that's to be expected.

Background For the Problematic Type Parameter

The generated OData client relies on an HTTP client which is configurable: Axios, Fetch, JQuery, ....roll your own. Each HTTP client comes with its very own configuration options. And each request should be configurable. Using the generic type here allows me to offer the correct typing for the configuration options:

export abstract class EntitySetServiceV4<
  ClientType extends ODataHttpClient,
  T,
  EditableT,
  Q extends QueryObject,
  EIdType
> {
  public async query<ReturnType extends Partial<T> = T>(
    queryFn?: (builder: ODataQueryBuilderV4<Q>, qObject: Q) => void,
    requestConfig?: ODataHttpClientConfig<ClientType> // here the correct config object is inferred from the HTTPClient
  ): ODataResponse<ODataCollectionResponseV4<ReturnType>> {
    // ....
  }

}

See source code

I don't know how to define this differently...

Is this a Bug?

Let me ask naively: Do you think that's bad / imperformant code that I'm using here? Should I have known better?

To ask the other way around: Do you accept this a TS bug?

I think one can argue for both positions. However, I would argue for the latter one: That TS uses recursion to evaluate variance is an implementation detail that I, as end user, am not aware of & probably shouldn't be. Code that works perfectly fine in smaller examples, breaks for bigger code bases. But just my 2 cents...

@ahejlsberg
Copy link
Member

I think this can reasonably be called a design limitation. Technically, there's nothing wrong with your code, but the same can be said for code that, for example, nests if statements to multiple thousand levels, or some other very deep construct that naturally doesn't occur in code authored by humans. All such code has the potential to overflow the call stack. We do have special handling for a few cases, such as our "trampoline" implementation of binary expression type checking, motivated by allowing concatenation of 1000s of string literals in a single expression. But externalizing the call stack through a trampoline comes with performance and complexity penalties, and it would be particularly hard to implement in this case because we'd have to completely revise our type relationship logic. Since the issue you're experiencing is rare (it's the first report I know of) and likely only occurs in machine generated code, a fix won't be a high priority.

You may want to experiment with adding in or out variance annotations for the ClientType type parameter. It will likely lead to errors, but you might be able to work around them. It is unfortunate that we don't have the ability to annotate bi-variance. If anything, that would be something to consider.

@texttechne
Copy link
Author

Ok, I was afraid you would argue that way, unfortunately (for me) reasonable enough 😄

Before closing this issue: Do you see any problem with adding // @ts-nocheck to the beginning of each file and leaving the code as is?

@texttechne
Copy link
Author

Finally found the time to toy around with variance annotations. After changing a certain implementation detail, I was able to annotate with out as well as in out and it works. The recursive variance computation is not triggered and the dreaded max call stack size exceeded error won't occur. Not as performant as leaving out the generic type parameter altogether, so the compilation will eventually run into the much more comfortable out of memory exception, but only for the 20MB metadata example and this is easily solved by giving the process more memory.

This is the best result as I'm now able to compile even those large examples which in turn brings any bugs with my generator to light.

But to be honest, that's more like trial and error for me than anything else. I think I got some hold of the variance topic in the meantime, but I fail to make the connection to my code. In the end I chose to annotate invariance...
However, not your responsibility to teach me stuff.

@ahejlsberg thank you so much for your analysis and all the right pointers! I was really lost there and needed help, which you provided brilliantly!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Needs More Info The issue still hasn't been fully clarified
Projects
None yet
Development

No branches or pull requests

4 participants