Moving providers out of tree

[Raffo]

Over the course of many years, we have added a lot of providers, some of which are not maintained or developed. This creates the expectation on users that we can fix everything or make those providers well supported. With the introduction of the webhook mechanism we stopped adding new providers, which has given other companies and DNS providers the possibility to integrate with ExternalDNS without having to add new code in tree, essentially also decoupling them from this project's release cycle.
The provider in tree also bring other challenges like the one of the very frequent dependency updates due to the update of the required libraries.
I propose that we start moving all the providers out of tree, with the exception of the stable providers, for now. I'd love to start with the unmaintained one, then move by group until we reach the AWS, Azure and Google one which are the most maintained ones and we should understand the impact on the usability of the project or even on cloud provider offerings before moving them out.

[Raffo]

Notifying @seanmalloy @vinny-sabatini @alejandrojnm @saileshgiri @Sh4d1 @packi @assureddt @hughhuangzh @Hyzhou @michaeljguarino @tinyzimmer who are listed as maintainers/contributors of existing providers.

[hughhuangzh]

@Raffo what's the standard about the stable providers, and unmaintained one?

[Raffo]

@hughhuangzh what do you mean by "Standard"? Can you clarify your question?

[hughhuangzh]

@Raffo I mean how to judge the stable providers.

[szuecs]

@hughhuangzh https://github.com/kubernetes-sigs/external-dns?tab=readme-ov-file#status-of-providers

[hans-m-song]

Hi, given the webhooks will be the primary mechanism moving forward, could there be some consideration for #4230? Would be much appreciated.

[Raffo]

@hans-m-song thanks for the nudge to link to the discussion, I definitely didn't see that there, the discussions/issues dual source isn't the easiest to deal with 😅 let me answer you there.

EDIT: asked to convert the discussion there to an issue with a proposal.

[costinm]

Any chance the 'out of tree' providers can be set as additional containers at install ( by adding a helm install option with the image ), to avoid all the mess with setting up certificates, keys and maintaining/upgrading 2 different apps - in particular if any change is made in the API ?

Also it may be worth defining how stable is the webhook API - there are standards on how to encode DNS zones and entries, broadly adopted - it may be worth using it as part of both the webhook API and DNSEndoint - the naming in the API is not very aligned with DNS terms and more close to K8S.

[Raffo]

Any chance the 'out of tree' providers can be set as additional containers at install

Where do you think the code for those should be living? I considered several approaches, but if we keep all the source in this repo, even if we stop linking them in the same binary of external dns we are not solving anything, just making life more complicated for users.

Also it may be worth defining how stable is the webhook API

This is of course a requirement before moving anything out of tree. At the time of writing, we're still considering the webhook as "experimental" although we've seen quite some adoption which was successful as much as I can say. Note that we collect zero analytics on installations, so we don't really have tons of data on our users and we have to work with the info that we have.