After update zone update fails. After getting notified the slave often reports 'Unable to retrieve SOA for' and no AXFR happens

[ramuller]

We use PowerDNS with 1 authoritative master with 5 authoritative slaves the slave also use dnsdist for load balancing.
I have updated PowerDNS from version 4.1.14 to 4.8.4. After the update the zone-transfer to the slave is very unreliable.
I can see the NOTIFY in the slave but at least half of the time I get "Unable to retrieve SOA for" and no AXFR is initiated.

Example for OK and NOK

Dec 12 12:28:19 dns-slave.example.net pdns_server[863666]: Received NOTIFY for foo-test.example.net from 127.0.0.1<-1.2.3.4/32 - queueing check
Dec 12 12:28:21 dns-slave.example.net pdns_server[863666]: Unable to retrieve SOA for foo-test.example.net, this was the first time. NOTE: For every subsequent failed SOA check the domain will be suspended from freshness checks for 'num-errors x 60 seconds', with a maximum of 3600 seconds. Skipping SOA checks until 1734006561
Dec 12 12:28:21 dns-slave.example.net pdns_server[863666]: Got NOTIFY for foo-test.example.net, removing zone from list of failed slave-checks and going to check SOA serial
Dec 12 12:28:22 dns-slave.example.net pdns_server[863666]: Received NOTIFY for foo-test.example.net from 127.0.0.1<-1.2.3.4/32
Dec 12 12:28:22 dns-slave.example.net pdns_server[863666]: Received NOTIFY for foo-test.example.net from trusted-notification-proxy 127.0.0.1<-1.2.3.4/32
Dec 12 12:28:22 dns-slave.example.net pdns_server[863666]: Received NOTIFY for foo-test.example.net from 127.0.0.1<-1.2.3.4/32 - queueing check
Dec 12 12:28:24 dns-slave.example.net pdns_server[863666]: Domain 'foo-test.example.net' is stale, master 1.2.3.4 serial 2024124065, our serial 2024124063

The network connection is stable and no additional firewall except nftables is between the nodes.
Even with loglevel 9 I don't get more detailed.
Has anyone else seen similar problems and good ideas how to debug the situation?

[Habbie]

Please:

• read Open Source Support: out in the open #13200
• then update your post with real log output
• see if the logs on the primary show anything interesting

[Habbie]

That might explain that the failed case does not see a SOA query to the one master you were monitoring with tcpdump.

[ramuller]

Yep looks like that's the problem. Is there a way to stick the SOA request to the source of notification?

[Habbie]

no, we pick a random primary from the set

[ramuller]

From the master column in the domain table?
Do I understand that correct?

[Habbie]

yes