domain name with trailing dot fails the certificate check when using gnutls

[ryandesign]

When built using gnutls, curl cannot connect to https://pyropus.ca. possibly because the period at the end of the hostname is not being stripped for SNI purposes. This problem does not occur when curl is built using openssl. I'm not sure whether this is a curl bug or a gnutls bug.

For prior history on issues accessing this server, which intentionally uses a canonical name ending in a period, see #8290 reported by @ccazabon. There, the problem was that curl was stripping the period from the hostname for non-SNI purposes when it should not have been doing so.

I did this

curl -I https://pyropus.ca./

I expected the following

HTTP/1.1 200 OK
Date: Fri, 19 Apr 2024 16:21:47 GMT
Server: Apache/2.4.57 (Debian)
Accept-Ranges: bytes
Vary: Accept-Encoding
X-Frame-Options: sameorigin
Content-Type: text/html

but I got this

curl: (60) SSL: certificate subject name (pyropus.ca) does not match target host name 'pyropus.ca.'
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

curl/libcurl version

curl 8.7.1 (x86_64-apple-darwin21.6.0) libcurl/8.7.1 GnuTLS/3.7.10 zlib/1.3.1 brotli/1.1.0 zstd/1.5.6 libidn2/2.3.7 libpsl/0.21.5 nghttp2/1.61.0 ngtcp2/1.4.0 nghttp3/1.2.0
Release-Date: 2024-03-27
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Largefile libz NTLM PSL SSL threadsafe TLS-SRP UnixSockets zstd

operating system

Darwin hostname-redacted 21.6.0 Darwin Kernel Version 21.6.0: Mon Feb 19 20:24:34 PST 2024; root:xnu-8020.240.18.707.4~1/RELEASE_X86_64 x86_64

[bagder]

I think this description is slightly off. The correct SNI field (without trailing dot) is actually used. The problem is rather in this call:

curl/lib/vtls/gtls.c

Line 1202 in 5fa594a

rc = gnutls_x509_crt_check_hostname(x509_cert, peer->hostname);

... where peer->hostname points to pyropus.ca. and GnuTLS returns 0, meaning failure.

[ryandesign]

I think this description is slightly off.

Likely so; I was guessing. Feel free to retitle or edit as needed.

[icing]

Please see #13440 for the fix and added test cases.

[icing]

Fix and tests now in #13486.