Reverse the KubeOpsSkipCrds... logic to support custom resource libraries?

[jefflill]

We've started to integrate your SDK into a multiple project solution and we'd like to implement a library of resources that can be shared by multiple projects. Here's how this is working now:

1. We create a Resources library project that references KubeOps and is where we define our custom resources. We add the <KubeoOpsSkip...>true</KubeoOpsSkip...> elements to the library's .csproj to prevent your build targets from trying to execute the library to perform generate the CRDs, etc. (which won't work of course, because it's just a library, not a console app).

2. We currently implementing two operators, one that runs as a daemonset on all cluster nodes and the other as a regular deployment. I have one of these configured to generate CRDs (via your build targets) and we have these installed by Helm when we install this operator into the cluster. The other operator and any subsequent operators won't generate or install CRDs.

3. Other (non-operator) projects also reference the Resources library to add and manage custom resources to the cluster.

The problem we're seeing is that your build targets are executed for every project that references KubeOps, even those with indirect references. This means that we need add <KubeoOpsSkip...>true</KubeoOpsSkip...> elements to every library or console project that references anything else that references KubeOps, which is a hassle and would be inconvenient for any innocent developer who wanted to import a custom resource library via nuget.

I wonder if it would be possible to reverse your logic to allow people to explicitly opt-in (rather then opting-out) to the CRD, etc... generation via something like?

    <KubeOpsDockerfile>true</KubeOpsDockerfile>
    <KubeOpsCrds>true</KubeOpsCrds>
    <KubeOpsRbac>true</KubeOpsRbac>
    <KubeOpsOperator>true</KubeOpsOperator>
    <KubeOpsInstaller>true</KubeOpsInstaller>

...and then update your project templates to include this.

The nice thing about this approach would be that it'll still just-work for operator projects and library and non-operator projects wouldn't have to do anything special to prevent the generator calls from breaking the build.

[buehler]

Hey @jefflill

This could be an option yes! What do others think about this change? (since it would be a breaking change)
/cc @tomkerkhove @erin-allison for example :)

[buehler]

oh, and of course /cc @anekdoti :)

[erin-allison]

I like the idea behind the change, and I had thought of a different way to solve this previously. A tool package could be built that contains the generator logic, very similar to dotnet-ef, since most of the time, you wouldn't necessarily need to ship the code to build your CRDs, they would be generated at build-time and be shipped alongside the binaries.

What could (in theory) be done is to break that logic out and use the same workflow as dotnet-ef does (dotnet/aspnetcore#31861) to avoid having to change the code surrounding the entrypoint.

[jefflill]

KubeOps ROCKS! I got my first production operator working yesterday using KubeOps and .NET.

One additional related idea:

It would be nice if the CRD related classes and attributes like:

• KubernetesEntityShortNamesAttribute
• EntityScopeAttribute
• DescriptionAttribute
• RequiredAttribute
• PatternAttribute
• CustomKubernetesEntity
• etc...

were published as separate netstandard2.0 nuget package.

This would be much cleaner for folks writing libraries with custom resource classes (like us). It's a bit weird to force all consumers of these libraries to also be .NET 6 (although this is the future of course).

[buehler]

Thank you very much @jefflill. I'm happy you enjoy it :-)

Yes, those elements could definitely be exported into an extra core library.

[buehler]

I just investigated some of the options for extracting a core package. There is no more "standard" in .net 5 and .net 6. All libraries and other packages combine .netstandard with the "full" .net. So I'll be able to extract a core package but I need to investigate if there will be a .netstandard package (due to the fact that I may use code from .net 6)

[jefflill]

That's not entirely true.

.NET Standard 2.0 and 2.1 libraries still work for .NET 5/6 and there are still a lot of people with legacy .NET 3.1 apps as well.

For example, the KubernetesClient nuget package actually includes .NET Standard 2.0 in addition to .NET 5 and .NET 6 versions. The nice thing here all legacy application from, .NET 3.1 and later are all supported.

https://www.nuget.org/packages/KubernetesClient/

So there are actually two reasons for splitting the custom resource related classes into a separate .NET Standard 2.1 library:

1. Support for legacy apps (discussed above)

2. Lower overhead for client apps

The problem here is that your KubeOps package includes all of the ASP.NET code which doesn't really make sense for client (non-operator) apps that are just making calls to the API server.

Imagine the extreme case of an iPhone app written in MAUI that just needs to retrieve some cluster status. It would be really unfortunate to require the phone app to include the entire ASP.NET stack just to be able use the Kubernetes client to query an API server.

Over at neonKUBE, we have a similar situation with a command line client that needs to perform operations on custom resources handled by an operator implemented with your SDK. That tool ends up with ASP.NET even though it's never used.

[erin-allison]

An addition to what's listed here - in case some components unavoidably touch some of the ASP.NET stuff, we should be able to import the ".Abstractions" package to get access to the interfaces inside a core library without having to include the bulk of the ASP.NET library into every copy of said library.

[buehler]

I stand corrected, very good point! I'll make a new "next" branch where we can move the core parts that are not part of the operator itself. A quick look over the code resulted mostly in interface definitions and entity stuff. The effective controller logic and reconnection logic to Kubernetes itself shall remain in the SDK because that's the part that actually communicates with Kubernetes when a software becomes an Operator. wdyt?

[jefflill]

Perfect!

[buehler]

One thing that came up: if we split the core out of the operator and publish it as .net6 and .netstandard2.1 lib, the generator must reside in the operator sdk library. The code generator (yaml stuff) cannot be separate since it needs access to the code in the project. It uses Assembly scanning to fetch types with the KubernetesEntity attribute to generate the CRDs and other things.

One possibility to solve this with an external tool would be to load the compiled assembly of the operator. Correct?

[erin-allison]

It would be half of it.

KubeOps would contain the parts that poll the K8S API, the event loop, etc for what is required for the bare minimum of an operator, and KubeOps.Web would build on top of that to add in the parts of KubeOps that right now depend on ASP.NET (inbound webhooks, healthchecks, etc)

[buehler]

Oh, I see. So in theory, one could build an operator without the help of ASP.net with the KubeOps package. And you are right: the only dependencies to ASP.net are the external available parts (webhooks). Back when I started the project I used ASP.net because it was a very convenient way to let an application "just run".

[erin-allison]

Also back then, use of the generic host was a bit more dependent on ASP.NET versus it being pretty well separated now.

[erin-allison]

Oh, I see. So in theory, one could build an operator without the help of ASP.net with the KubeOps package.

And yes, you're correct. If I just want something simple to watch a set of entities on an interval, I don't need the bulk of ASP.NET brought into my application. I just need pretty much a single hosted service and a handful of dependencies.

[buehler]

That is a well thought architecture. Thank you for the input!

[buehler]

Hey all :)

Has been a while. Hope you all are doing well!

I create a new branch to work on when I got the time: https://github.com/buehler/dotnet-operator-sdk/tree/refactor/new-setup

What do you think about the setup? This would result in more packages, however, the possibility exists to create operators without web components (asp) and use the generators outside the same lib. The core should be one lib (.netstandard or .net 6/7/8, don't know yet).

/cc @erin-allison @jefflill