Codegen with TypeSpec

[jlarmstrongiv]

Clear and concise description of the problem

@bterlson mentioned in hackernews that:

Codegen is coming online as we speak. We do codegen from TypeSpec in Azure across multiple languages, and the results are pretty great. We're moving that over to the TypeSpec project so everyone can generate code. Obviously my opinion is biased, but I think the results are significantly better than what you find elsewhere in the ecosystem.

I also watched @garethj-msft Nordic API’s presentation “Reigniting the API Description Wars with TypeSpec and the Next Generation of API Languages” where he described the history leading to TypeSpec and mentioned Microsoft’s Kiota project

There is also an existing PR to add a csharp emitter #2458

My question is whether I should plan to use TypeSpec to generate OpenAPI Specification to use with Kiota, or if I should wait for codegen to be added to this repo. If codegen will be added to this repo soon, then this issue may be a good place to track its release.

Anyway, I look forward to trying out TypeSpec and friends next week! Thank you for open-sourcing it.

Checklist

• Follow our Code of Conduct
• Read the docs.
• Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

[bterlson]

This is a great discussion topic, but I will share a tracking issue here, that's a good idea.

It's hard to say which you should do, it depends a lot on what your needs are and how urgent those needs are.

Kiota codegen is good, so I would say, if it works for you, then you would have no problems going the openapi -> kiota route. A side project I hope to get to soon is to package this up in an emitter to make it even easier, so you could do something like npx tsp compile . --emit typespec-kiota --option typespec-kiota.language="csharp"). That said, I am obviously partial to the codegen we do on the Azure SDK and are bringing to TypeSpec. Kiota codegen wouldn't work for us, so if you're in a similar boat, I would recommend waiting. Try it and see what you think!

In terms of timeline, we have alpha quality generators now that you can try (see https://github.com/bterlson/typespec-todo). I expect to see codegen come fully online with the rest of the project (high quality beta, with a fairly high bar for breaking changes) over the next 3 months.

[attilah]

Don't know if carrying on this discussion or opening a new one would be better, but as the title is generic, I just put it here. Let me know if you would want it in a new discussion.

How I look at TypeSpec is capable much more than just to define things around the "Api realm" and enables much broader scenarios to generate code and others with custom models and emitters.

In the recent days I looked through a lot of publicy available resources, packages, code regarding TypeSpec (like: Azure tooling, AutoRest, MS Graph package) and this takes me to my question:

Other MS teams seems to heavily leverage the TypeSpec typing system to (re)define their own types and not going with the built-in pieces provided by Rest and Http TypeSpec modules.

A dozens of decorators are implemented and for me the only way to take those to the CodeModel would be possible only by doing a custom CodeModel and change/extend the emitter code like those teams are doing. It is quite an involved process that would be done all the TypeSpec users around the globe.

Example scenario:

I want to add a @immutable decorator to my TypeSpec library that could by applied on ModelProperty nodes and this needs to be available in the CodeModel to be able to generate code from it.

What are the extensibility thoughts/plans regarding the support of such feature without inheriting from the CodeModel class and extend it or define our own CodeModel whenever it is needed?

When it comes to code generation, I prefer to leverage the target languages (C# in this case) tools (Roslyn) and not using some other languages (Java/Kotlin in case of Smithy, TypeScript in case of TypeSpec).

[bterlson]

I'll share my personal thoughts on these questions. The CodeModel approach exists I think for two main reasons:

1. Because we don't have compiler RPC, emitters have to serialize the parts of the type graph they care about, so in practice some representation aside from the type graph has to exist
2. Because Azure client codegen is fairly sophisticated (e.g. internally we have the ability to customize client codegen from tsp), and much of that sophistication relies on the relatively complex interaction of a number of type declarations and decorators, leading to a need to share the logic to understand these details between emitters which want to understand them.

The first constraint is a point in time thing - we want to support an RPC model that allows querying the type graph directly. The second constraint is perhaps unique to our internal emitters, and so we don't consider the CodeModel a part of TypeSpec's public surface area per se, more of a sausage making detail of our emitters. Over time, the Azure-specific parts of our emitters will be moved out of this repository, and along with it perhaps the CodeModel as well. Also, the compiler APIs continue to improve, providing more and more APIs that make consuming the type graph easier.

My personal opinion is that in general a shared CodeModel approach is untenable outside of a very constrained emitter domain like "Azure REST clients". As the number of scenarios grows (different languages, different customizations, different client/service types, different protocols, etc.) the CodeModel approaches or exceeds the complexity of the graph itself, and so you might as well just use the graph (lack of an RPC interface notwithstanding). So I don't think having a shared CodeModel as part of the TypeSpec project is even that useful, and prefer an approach where the compiler APIs make consuming the graph directly easier. But I could very well be wrong with this approach and look forward to learning more as folks try out TypeSpec and share more of the ways they're using it!

One final point is even with some RPC interface to the compiler, I think it will always be preferrable to author emitters in TypeScript. RPC will have a fair amount of overhead, but the emitter framework is one of the biggest reasons. We are presently working on a revision to the emitter framework that will make writing emitters substantially easier, and also make things like composing emitters far less of a challenge (think things like plugging in custom model emitters, swapping out the operation emitter to generate Azure Function stubs, or customizing codegen like we do for Azure). I don't anticipate we would provide similar framework capabilities across all languages, although the community could perhaps build them.

[attilah]

Thanks Brian for the elaborate explanation, and I am on the same page with you and the necessity of a CodeModel is a good thing, it is like an AST -> AST transformation from TypeSpec to CodeModel, which is futher transformed into another AST in a case where someone uses Roslyn to generate the code.

Lets treat "Azure" as a domain of MSFT, and in my case my line of business API is my domain and because of the business related complexity and it's specifics will require me to have a custom CodeModel so I can articulate the domain features within it and use it further.

Based on this I think with a mindset shift on how we look at the CodeModel, what if we would treat the CodeModel approach as a recipe and instead of removing it in the future from TypeSpec, turn it into patterns and practices and guidance (pun intended, @garethj_msft will know) on how someone could roll their own, what steps are needed, etc.

Beside having it as guidance, as long as it is only a bunch of related POCO classes, perhaps a way could be found on how to generate these from the TypeSpec files (as currently it is generated from JSON Schemas for Azure).

I think that your opinion that you've formulated here - even if personal - it determines how deep or what way one would invest into TypeSpec ecosystem. For me based on this it means to get extract out the CodeModel schemas and generation and related code and start to use that so save myself from future surprises when it will be just gone. This looks the safest and most future proof way for my journey. Please correct me if I miss something.