SSLPeerUnverifiedException Hostname fd33:1a73:fa8f::1 not verified after upgrade to Boot 3.4.0 and Cloud 2024.0.0

[mle-idealo]

Describe the bug
After the upgrade from Spring Boot 3.3.6 to 3.4.0 and Spring Cloud 2023.0.4 to 2024.0.0 we're experiencing issues regarding hostname verification which read like

javax.net.ssl.SSLPeerUnverifiedException: Hostname fd33:1a73:fa8f::1 not verified:
    certificate: sha256/bLcj0Q+HP/EF+4njk0xrQvqb/KtOHnZa2xf+rl9ldkc=
    DN: CN=kube-apiserver
    subjectAltNames: [fd33:1a73:fa8f:0:0:0:0:1, 2a05:d014:396:cd05:0:0:0:e781, 172.16.98.175, 55c2d4e83b3377534d8c22d619c3cb94.gr7.eu-central-1.eks.amazonaws.com, ip-172-16-98-175.eu-central-1.compute.internal, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local]
	at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:334)
	at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:284)
	at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:169)
        [...]
	at java.lang.Thread.run(Thread.java:1570)

Wrapped by: java.io.IOException: Hostname fd33:1a73:fa8f::1 not verified:
    certificate: sha256/bLcj0Q+HP/EF+4njk0xrQvqb/KtOHnZa2xf+rl9ldkc=

    DN: CN=kube-apiserver
    subjectAltNames: [fd33:1a73:fa8f:0:0:0:0:1, 2a05:d014:396:cd05:0:0:0:e781, 172.16.98.175, 55c2d4e83b3377534d8c22d619c3cb94.gr7.eu-central-1.eks.amazonaws.com, ip-172-16-98-175.eu-central-1.compute.internal, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local]

	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:504)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:524)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleGet(OperationSupport.java:467)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleGet(BaseOperation.java:792)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.requireFromServer(BaseOperation.java:193)
	... 20 common frames omitted

Wrapped by: io.fabric8.kubernetes.client.KubernetesClientException: Operation: [get]  for kind: [Pod]  with name: [offer-attribute-assignor-6778d89688-pdm8h]  in namespace: [offer-attribute-assignor]  failed.
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:159)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.requireFromServer(BaseOperation.java:195)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.get(BaseOperation.java:149)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.isReady(BaseOperation.java:919)
	... 5 frames excluded
	... 13 common frames omitted

Wrapped by: org.springframework.context.ApplicationContextException: Failed to start bean 'leaderInitiator'
	at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:326)
	at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:510)
	at java.lang.Iterable.forEach(Iterable.java:75)
	... 10 frames excluded
	at de.idealo.orca.attribute.ApplicationKt.main(Application.kt:24)

There are various bug reports similar to this one available but they all date years back and are already closed.
I suspect the crucial change anywhere within io.fabric8:kubernetes-client between versions 6.9.2 (Cloud 2023.0.4) and 6.13.4 (Cloud 2024.0.0).

From my understanding the address fd33:1a73:fa8f::1 should be fine regarding verification.
Did we miss something?

[ryanjbaxter]

Can you try setting the environment variable KUBERNETES_DISABLE_HOSTNAME_VERIFICATION_SYSTEM_PROPERTY or setting the system property kubernetes.disable.hostname.verification to true?

I am pretty sure this change altered the behavior.
fabric8io/kubernetes-client@08b0e9f#diff-7506ddc44edef04cb34dcde13c56d00ed3b1d35d848ded8f1924ed44c03db953R403

[mle-idealo]

Unfortunately despite set env variable, no change in behavior. The host name is still getting verified:

"stack_trace":"javax.net.ssl.SSLPeerUnverifiedException: Hostname fd33:1a73:fa8f::1 not verified
at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:334)

I've double-checked this also by setting the respective system property instead of the env var, same result, host name is still getting verified.

[ryanjbaxter]

Sorry looks like the environment variable should be KUBERNETES_DISABLE_HOSTNAME_VERIFICATION. Can you provide a sample that reproduces the issue?

[mle-idealo]

Unfortunately I am off until Monday but regarding the env var, one of my colleagues, @andibraeu, could give this a try. However apart from the slightly renamed env var name, should not also the system property you mentioned yield the same effect?

Regarding the sample, I could provide one next week. I'll just shrink our project down, since this popped up in many repos and they're not that fancy, just some Kafka Listeners and a MongoDb.

[wind57]

It could actually be us here... The code used to do:

Inet4Address.getLocalHost().getHostName();

and now it does:

InetAddress.getLocalHost().getHostName();

I don't know this API too much, but I will do some reading and come back

off-top: hats off for using Shenandoah! :) we use it also in production for quite some time

[wind57]

It's a bit weird that you do not have the env variable HOSTNAME in the pod btw. can you issue an env inside the pod? because our code does this:

	public static String hostName() throws UnknownHostException {
		String hostName = EnvReader.getEnv(HOSTNAME);
		if (StringUtils.hasText(hostName)) {
			return hostName;
		}
		else {
			return InetAddress.getLocalHost().getHostName();
		}
	}

[ryanjbaxter]

But the stacktrace seems to point to the Fabric8 code, I don't see anything pointing at InetAddress...

[wind57]

ah, indeed, you're right.

[mle-idealo]

Sorry looks like the environment variable should be KUBERNETES_DISABLE_HOSTNAME_VERIFICATION. Can you provide a sample that reproduces the issue?

Wow, that did the trick. With this env var the update to Boot 3.4.0 and Cloud 2024.0.0 was successful.

So two questions that arise on our side:

1. What do we now lose with disabled hostname verification?
2. What change on your side resulted in this changed behavior? Did we miss something in the release notes?

[ryanjbaxter]

I am not a network expert here but I am having a hard time explaining this.

This ip fd33:1a73:fa8f::1 is the shortened form of fd33:1a73:fa8f:0:0:0:0:1 which is listed in the subjectAltNames of the certificate.

OKHttp takes this into account in its hostname verifier
https://github.com/square/okhttp/blob/parent-4.12.0/okhttp/src/main/kotlin/okhttp3/internal/tls/OkHostnameVerifier.kt#L61
Assuming it is even getting there https://github.com/square/okhttp/blob/4984568367caaf359b82c452bd28b5e192824d1c/okhttp/src/main/kotlin/okhttp3/internal/tls/OkHostnameVerifier.kt#L54

Maybe one thing to try would be to add fd33:1a73:fa8f::1 to the certificate and see if that makes things work. But to be honest this is an OKHTTP question, but my guess is there is something with your certificate that is causing an issue.

[ryanjbaxter]

@wind57 not sure if you see something I don't here

[wind57]

a little caught up with other things, but will take a look soon

[spring-cloud-issues]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.