Cloudflare DoT w/multiple requests = DNSSEC fail until connection closed

[uzlonewolf]

So, I don't know if this is a Cloudflare, Stubby, or GetDNS bug, but when Stubby is configured to use only Cloudflare and multiple requests for the same name are sent, both of those requests plus every request sent after that point will fail. Things only start working again once the connection has been idle long enough to be closed.

Using this configuration:

resolution_type: GETDNS_RESOLUTION_STUB
dns_transport_list:
  - GETDNS_TRANSPORT_TLS
tls_authentication: GETDNS_AUTHENTICATION_REQUIRED
tls_query_padding_blocksize: 128
edns_client_subnet_private : 1
round_robin_upstreams: 0
idle_timeout: 10000
tls_min_version: GETDNS_TLS1_2
listen_addresses: 
  - 127.0.0.1@9953
  - 0::1@9953
dnssec: GETDNS_EXTENSION_TRUE
appdata_dir: "/var/cache/stubby"
dnssec_trust_anchors: "/var/lib/unbound/root.key"
upstream_recursive_servers:
## Cloudflare 1.1.1.1 and 1.0.0.1
  - address_data: 1.1.1.1
    tls_auth_name: "cloudflare-dns.com"
  - address_data: 1.0.0.1
    tls_auth_name: "cloudflare-dns.com"

I can reliably get it to fall over with:

dig @0 -p 9953 google.com; dig @0 -p 9953 google.com & dig @0 -p 9953 google.com

Changing round_robin_upstreams: 0 to round_robin_upstreams: 1 makes it take slightly longer, but it still happens if you run the above 2-3 times.

Recompiling GetDNS with full debugging enabled shows that once the bug is triggered, every subsequent request over that TLS/TCP connection will result in a DNSSEC failure. Once that happens, Stubby/GetDNS just sits there and times out instead of retrying with the next server.

I have not tried DoH, but this does not happen with Google's DoT servers.

[hanvinke]

Strange behaviour indeed. How exactly does it fall over? Just AD-bit not showing in the response? Does the same thing also happen when you use 'dnssectest.sidn.nl' instead of 'google.com' ?

I am just asking because 'google.com' does not give me the AD-bit in return (I have a similar configuration). But 'dnssectest.sidn.nl' does and answers more quickly.

[uzlonewolf]

I'm not sure exactly, getdns just says DNSSEC validation failed and drops the response. google.com is just an example, it does it for every name. Here is a portion of the rather verbose debug logs when using dnssectest.sidn.nl:

[11:20:05.840805] Validating dnssectest.sidn.nl. IN A, rrs: 1, rrsigs: 1
[11:20:05.840829] 	with trust anchor . IN DNSKEY, rrs: 1, rrsigs:
[11:20:05.840861] Commencing NX proof for: dnssectest.sidn.nl. IN DS, rrs:, rrsigs:
[11:20:05.840902] Commencing NX proof for: sidn.nl. IN DS, rrs:, rrsigs:
[11:20:05.840945] Commencing NX proof for: nl. IN DS, rrs: 1, rrsigs: 1
[11:20:05.841026] sizes: 2 rrs, 569 bytes for validation buffer
[11:20:05.841056] written to valbuf: 569 bytes
[11:20:05.841476] key .	86400	IN	DNSKEY	257 3 8 AwEAAaz/tAm8yTn4Mfeh5eyI96WSVexTBAvkMgJzkKTOiW1vkIbzxeF3+/4RgWOq7HrxRixHlFlExOLAJr5emLvN7SWXgnLh4+B5xQlNVz8Og8kvArMtNROxVQuCaSnIDdD5LKyWbRd2n9WGe2R8PzgCmr3EgVLrjyBxWezF0jLHwVN8efS3rCj/EWgvIWgb9tarpVUDK/b58Da+sqqls3eNbuv7pr+eoZG+SrDK6nWeL3c6H5Apxz7LjVc1uTIdsIXxuOLYA4/ilBmSVIzuDWfdRUfhHdY6+cn8HFRm+2hM8AnXGXws9555KrUB5qihylGa8subX2Nn6UwNR1AkUTV74bU= ;{id = 20326 (ksk), size = 2048b}
[11:20:05.841522] signed . IN DNSKEY, rrs: 1 2, rrsigs: 1
[11:20:05.841547] Commencing NX proof for: nl. IN DS, rrs: 1, rrsigs: 1
[11:20:05.841596] sizes: 1 rrs, 69 bytes for validation buffer
[11:20:05.841626] written to valbuf: 69 bytes
[11:20:05.841953] key .	172474	IN	DNSKEY	256 3 8 AwEAAeB54o2xvW6vY4qQZ0krDsEZCe6MsRWCqsXd4+cNJZMePnlV/xwDrIbbeH1SJzv742rOHzgAKM1/3SQHHSkoEIPx8XQdHAZBxfhaXl3e8c5WrE3aGXS5AeTWAkt85ccqWgKyitxjFmJEOol0BqS2xueltaDwgWcC10nPUY+y5l/kTOYyptYQS4gg1uJNXIob/R1XIEJ10ZCurkYqZxgqyHc7tZv09N23o9rnGdjnYiArH7FjlXD8Rvjde8YWkmfdbCEWnchrnxDK8KV2/ZvBpG/WYnRKXYPUceGCw59OJdJ5M7utkm547RB3eEd8CVVhbXopZlsKq3GCrBwaIVe9ci0= ;{id = 18733 (zsk), size = 2048b}
[11:20:05.841994] signed nl. IN DS, rrs: 1, rrsigs: 1
[11:20:05.842027] ds_authenticates_keys DS: nl. IN DS, rrs: 1, rrsigs: 1
[11:20:05.842055] ds_authenticates_keys DNSKEY: nl. IN DNSKEY, rrs: 1 2, rrsigs: 1
[11:20:05.842110] sizes: 2 rrs, 450 bytes for validation buffer
[11:20:05.842131] written to valbuf: 450 bytes
[11:20:05.842453] key nl.	1442	IN	DNSKEY	257 3 8 AwEAAcb+4kIsKoZM+3ZZpU9kzxrzw30e3b+L0KZeX+aAS3eM+Q+q27Jw0NZ3dqsPSif61GjRW6apjDZ9Ciab3oyEu7IpihVrw94DTjWZTVViZAijAIHwKUzY0YjkT3RvN+xgpw4uZs1SnqCZxYko+15esteKXW/nJpde0d9OeFFBaS2WTCycK+A6gd9DsOw91Y7Z2vrR/2g9N9dMIVq9neB1/KXXm4MttLqJyxRWZNAFTyLGQKzPpQDp9s3qowV2+pcHOh6lUTEeOWiAtotJ/5WyO91viZ5tBfClsyGpggBTaeUQ7T5adhAtX6nRkhePyAtQgCCf63ZpHyoyxvbkDM7yuA0= ;{id = 34112 (ksk), size = 2048b}
[11:20:05.842496] signed nl. IN DNSKEY, rrs: 1 2, rrsigs: 1
[11:20:05.842531] keyset authenticated: nl. IN DNSKEY, rrs: 1 2, rrsigs: 1
[11:20:05.842547] valid_dsses: 1, supported_dsses: 1
[11:20:05.842570] Commencing NX proof for: sidn.nl. IN DS, rrs:, rrsigs:
[11:20:05.842609] Commencing NX proof for: dnssectest.sidn.nl. IN DS, rrs:, rrsigs:
[11:20:05.842650] Validating dnssectest.sidn.nl. IN A, rrs: 1, rrsigs: 1
[11:20:05.842677] 	with keys nl. IN DNSKEY, rrs: 1 2, rrsigs: 1
[11:20:05.842727] BOGUS: dnssectest.sidn.nl. IN A, rrs: 1, rrsigs: 1
[11:20:05.842748] 	with trust anchor: . IN DNSKEY, rrs: 1, rrsigs:

Edit: and here's a successful single request for that domain:

[11:28:54.249983] Validating dnssectest.sidn.nl. IN A, rrs: 1, rrsigs: 1
[11:28:54.250007] 	with trust anchor . IN DNSKEY, rrs: 1, rrsigs:
[11:28:54.250047] Commencing NX proof for: dnssectest.sidn.nl. IN DS, rrs:, rrsigs:
[11:28:54.250099] Commencing NX proof for: sidn.nl. IN DS, rrs: 1, rrsigs: 1
[11:28:54.250157] Commencing NX proof for: nl. IN DS, rrs: 1, rrsigs: 1
[11:28:54.250212] sizes: 2 rrs, 569 bytes for validation buffer
[11:28:54.250245] written to valbuf: 569 bytes
[11:28:54.250650] key .	86400	IN	DNSKEY	257 3 8 AwEAAaz/tAm8yTn4Mfeh5eyI96WSVexTBAvkMgJzkKTOiW1vkIbzxeF3+/4RgWOq7HrxRixHlFlExOLAJr5emLvN7SWXgnLh4+B5xQlNVz8Og8kvArMtNROxVQuCaSnIDdD5LKyWbRd2n9WGe2R8PzgCmr3EgVLrjyBxWezF0jLHwVN8efS3rCj/EWgvIWgb9tarpVUDK/b58Da+sqqls3eNbuv7pr+eoZG+SrDK6nWeL3c6H5Apxz7LjVc1uTIdsIXxuOLYA4/ilBmSVIzuDWfdRUfhHdY6+cn8HFRm+2hM8AnXGXws9555KrUB5qihylGa8subX2Nn6UwNR1AkUTV74bU= ;{id = 20326 (ksk), size = 2048b}
[11:28:54.250698] signed . IN DNSKEY, rrs: 1 2, rrsigs: 1
[11:28:54.250722] Commencing NX proof for: nl. IN DS, rrs: 1, rrsigs: 1
[11:28:54.250770] sizes: 1 rrs, 69 bytes for validation buffer
[11:28:54.250788] written to valbuf: 69 bytes
[11:28:54.251109] key .	165104	IN	DNSKEY	256 3 8 AwEAAeB54o2xvW6vY4qQZ0krDsEZCe6MsRWCqsXd4+cNJZMePnlV/xwDrIbbeH1SJzv742rOHzgAKM1/3SQHHSkoEIPx8XQdHAZBxfhaXl3e8c5WrE3aGXS5AeTWAkt85ccqWgKyitxjFmJEOol0BqS2xueltaDwgWcC10nPUY+y5l/kTOYyptYQS4gg1uJNXIob/R1XIEJ10ZCurkYqZxgqyHc7tZv09N23o9rnGdjnYiArH7FjlXD8Rvjde8YWkmfdbCEWnchrnxDK8KV2/ZvBpG/WYnRKXYPUceGCw59OJdJ5M7utkm547RB3eEd8CVVhbXopZlsKq3GCrBwaIVe9ci0= ;{id = 18733 (zsk), size = 2048b}
[11:28:54.251155] signed nl. IN DS, rrs: 1, rrsigs: 1
[11:28:54.251194] ds_authenticates_keys DS: nl. IN DS, rrs: 1, rrsigs: 1
[11:28:54.251221] ds_authenticates_keys DNSKEY: nl. IN DNSKEY, rrs: 1 2, rrsigs: 1
[11:28:54.251275] sizes: 2 rrs, 450 bytes for validation buffer
[11:28:54.251301] written to valbuf: 450 bytes
[11:28:54.251623] key nl.	1237	IN	DNSKEY	257 3 8 AwEAAcb+4kIsKoZM+3ZZpU9kzxrzw30e3b+L0KZeX+aAS3eM+Q+q27Jw0NZ3dqsPSif61GjRW6apjDZ9Ciab3oyEu7IpihVrw94DTjWZTVViZAijAIHwKUzY0YjkT3RvN+xgpw4uZs1SnqCZxYko+15esteKXW/nJpde0d9OeFFBaS2WTCycK+A6gd9DsOw91Y7Z2vrR/2g9N9dMIVq9neB1/KXXm4MttLqJyxRWZNAFTyLGQKzPpQDp9s3qowV2+pcHOh6lUTEeOWiAtotJ/5WyO91viZ5tBfClsyGpggBTaeUQ7T5adhAtX6nRkhePyAtQgCCf63ZpHyoyxvbkDM7yuA0= ;{id = 34112 (ksk), size = 2048b}
[11:28:54.251665] signed nl. IN DNSKEY, rrs: 1 2, rrsigs: 1
[11:28:54.251699] keyset authenticated: nl. IN DNSKEY, rrs: 1 2, rrsigs: 1
[11:28:54.251723] valid_dsses: 1, supported_dsses: 1
[11:28:54.251756] Commencing NX proof for: sidn.nl. IN DS, rrs: 1, rrsigs: 1
[11:28:54.251807] sizes: 1 rrs, 77 bytes for validation buffer
[11:28:54.251826] written to valbuf: 77 bytes
[11:28:54.251988] key nl.	1237	IN	DNSKEY	256 3 8 AwEAAdBMHLVXYPJGb+o8QXIj9v1CtqTu+EybrWyG/ShJp8/tc37GBt5NR7DzugoE9nfr2M0eKjnc4E/qO6GauZouq6tLQuC8k4RD49u0/ecYZV/+EZ6Qv3I/iNOyatR9qhluj1V16hLRpMsHApVFE8rWoSlcxqM9Sbe4BfAHBmx582cp ;{id = 52584 (zsk), size = 1024b}
[11:28:54.252026] signed sidn.nl. IN DS, rrs: 1, rrsigs: 1
[11:28:54.252059] ds_authenticates_keys DS: sidn.nl. IN DS, rrs: 1, rrsigs: 1
[11:28:54.252087] ds_authenticates_keys DNSKEY: sidn.nl. IN DNSKEY, rrs: 1 2, rrsigs: 1
[11:28:54.252129] sizes: 2 rrs, 201 bytes for validation buffer
[11:28:54.252156] written to valbuf: 201 bytes
[11:28:54.252967] key sidn.nl.	2946	IN	DNSKEY	257 3 13 jh1pVS3yzgJk4lM9fEtwS5V+nbISmn8AmGZAdY0T1o4351IQEBd/v518tl00YmQL5Mq26YLE8RGKIOCXAYzWiQ== ;{id = 62949 (ksk), size = 256b}
[11:28:54.253044] signed sidn.nl. IN DNSKEY, rrs: 1 2, rrsigs: 1
[11:28:54.253079] keyset authenticated: sidn.nl. IN DNSKEY, rrs: 1 2, rrsigs: 1
[11:28:54.253098] valid_dsses: 1, supported_dsses: 1
[11:28:54.253128] Commencing NX proof for: dnssectest.sidn.nl. IN DS, rrs:, rrsigs:
[11:28:54.253172] Validating dnssectest.sidn.nl. IN A, rrs: 1, rrsigs: 1
[11:28:54.253202] 	with keys sidn.nl. IN DNSKEY, rrs: 1 2, rrsigs: 1
[11:28:54.253240] sizes: 1 rrs, 61 bytes for validation buffer
[11:28:54.253266] written to valbuf: 61 bytes
[11:28:54.254002] key sidn.nl.	2946	IN	DNSKEY	256 3 13 Mel453Ed98AIkFkU4/SW8hvlCB6SCqP8zNjhsVJWzXIj7TAxC+8CWY2FbgGK5pVe9hc4jrL+BET4UWJ+gJ4/WA== ;{id = 39816 (zsk), size = 256b}
[11:28:54.254047] signed dnssectest.sidn.nl. IN A, rrs: 1, rrsigs: 1
[11:28:54.254075] Key 105352 proved
[11:28:54.254105] 	SECURE: dnssectest.sidn.nl. IN A, rrs: 1, rrsigs: 1

[hanvinke]

I have also tried this on my openWRT device. It seems that there is always an answer either very slow or with a time-out. But I do have the AD-flag in the response.

dnssectest.txt

[hanvinke]

@uzlonewolf could you please try 'Zero configuration DNSSEC' with Stubby by disabling 'dnssec_trust_anchors', the 3 files (root-anchors.p7s ,root-anchors.xml and root.key) should appear in your appdata_dir.

Recently I had some troubles with 'dig' after an update and had to force-remove the depending ssl libraries and reinstall them.
ldd /usr/bin/dig gives me currently:

    /lib/ld-musl-aarch64.so.1 (0xffffb377a000)
    libisc-9.18.7.so => /usr/lib/libisc-9.18.7.so (0xffffb31e0000)
    libdns-9.18.7.so => /usr/lib/libdns-9.18.7.so (0xffffb2ffe000)
    libisccfg-9.18.7.so => /usr/lib/libisccfg-9.18.7.so (0xffffb2fbb000)
    libirs-9.18.7.so => /usr/lib/libirs-9.18.7.so (0xffffb2fa7000)
    libbind9-9.18.7.so => /usr/lib/libbind9-9.18.7.so (0xffffb2f82000)
    libc.so => /lib/ld-musl-aarch64.so.1 (0xffffb377a000)
    libuv.so.1 => /usr/lib/libuv.so.1 (0xffffb2f46000)
    libssl.so.1.1 => /usr/lib/libssl.so.1.1 (0xffffb2eb3000)
    libcrypto.so.1.1 => /usr/lib/libcrypto.so.1.1 (0xffffb2c49000)
    libz.so.1 => /usr/lib/libz.so.1 (0xffffb2c22000)
    libnghttp2.so.14 => /usr/lib/libnghttp2.so.14 (0xffffb2be6000)
    libns-9.18.7.so => /usr/lib/libns-9.18.7.so (0xffffb2b95000)
    libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xffffb2b70000)

[wtoorop]

I have tried this, but I have had mostly slow responses like @hanvinke had, But I did see a servfail at some point. With Quad8 I have slower responses, but not as slow as with Quad1. Quad9 never gives me a slow response in this configuration.

I'm not 100% sure, but I suspect there some kind of stricter rate-limiting with CF hampering the lookup of some of the records to do validation successfully. If that would be the case, some kind of caching mechanism (at least for the DNSSEC answers) would be the definite solution.

[hanvinke]

With DiG 9.18.10 there seem to be improvements in speed and handling. I get a much quicker response from dnssectest.sidn.nl and no time-outs so far.

dnssectest1.txt

// edit: still have slow responses or time-outs occasionally. A query to a non existing website gives me no SERVFAIL btw.

[uzlonewolf]

@hanvinke

With DiG

I'm just using DiG to demonstrate how to reproduce the problem. In normal everyday use it is my web browser which is sending the multiple requests thereby causing stubby/Cloudflare to fall over.

@wtoorop I'm actually using unbound to cache responses and redirect certain names, however if there is no cached entry then it sends everything (including multiple requests for the same name) straight through.