dnsdist how to use xsk by xdp.py

[serious-snow]

referenc https://dnsdist.org/advanced/xsk.html

when I exec
python3 xdp.py --xsk --interface enp2s0f1
it show can't find map
error: map not found:

Do I need to create these myself?

/sys/fs/bpf/dnsdist/addr-v4
/sys/fs/bpf/dnsdist/addr-v6
/sys/fs/bpf/dnsdist/qnames
/sys/fs/bpf/dnsdist/cidr4
/sys/fs/bpf/dnsdist/cidr6
/sys/fs/bpf/dnsdist/xskmap
/sys/fs/bpf/dnsdist/xsk-destinations-v4
/sys/fs/bpf/dnsdist/xsk-destinations-v6

I didn't find any more information. If I need to create it, use bpftool, is there any example?

[serious-snow]

but these are automatically created when I execute scripts in centos9

[rgacogne]

Thanks a lot for the feedback! I completely understand, no worries! If you don't mind me asking, what kind of performance did you achieve, and what did you expect/hope for?

[serious-snow]

I found a magical place, when kxdpgun qps=200w time=30s, the replies exceeded the requests.

total queries:     60000800 (2000026 pps) (100.001333%)
total replies:     63405646 (2113521 pps) (105.674668%)
average DNS reply size: 142 B
average Ethernet reply rate: 3174600960 bps (3174.60 Mbps)
responded NOERROR:   63405646
duration: 30 s

I said the performance is not as expected, I stress tested 100w 200w 300w 500w
If qps is less than or equal to 200w, the performance is better with xsk turned on, otherwise it is higher without turning it on.
If you have time, can you help me see if there is any optimization in my configuration file? With or without xsk turned on, my downstream node does not support bpf because it is written in golang. Another reason why I don't use it is that my machine may restart after turning on bpf, as mentioned before.

local downstreamServers = { "10.20.0.50", "10.20.0.51" }

 setUDPSocketBufferSizes(16777216,16777216)
 addCapabilitiesToRetain("CAP_SYS_ADMIN")
 addCapabilitiesToRetain("CAP_NET_ADMIN")

function addServers(i)
    for k, v in ipairs(downstreamServers) do
     newServer({address=v, name='dns'..k, checkInterval=1, checkType="ANY", checkName=".", useProxyProtocol=true})
    end
end
function addListeners(i)
    local xsk = newXsk({ifName="ens785f0", NIC_queue_id=i-1, frameNums=65536, xskMapPath="/sys/fs/bpf/dnsdist/xskmap"})
    addLocal("10.20.0.49:53", {reusePort=true,cpus={i-1},xskSocket=xsk})
end

 local bpf = newBPFFilter({ipv4MaxItems=10240,
  ipv6MaxItems=10240,
  qnamesMaxItems=1024,
  cidr4MaxItems=1024,
  cidr6MaxItems=1024,
 })
  setDefaultBPFFilter(bpf)
  local dbr = dynBlockRulesGroup() 
--  dbr:setQueryRate(200, 1, "Exceeded query rate", 20)
-- dbr:setRCodeRate(DNSRCode.NXDOMAIN, 20, 10, "Exceeded NXD rate", 60)
-- dbr:setRCodeRate(DNSRCode.SERVFAIL, 20, 10, "Exceeded ServFail rate", 60)
-- dbr:setQTypeRate(DNSQType.ANY, 5, 10, "Exceeded ANY rate", 60)
-- dbr:setResponseByteRate(10000, 10, "Exceeded resp BW rate", 60)

 function maintenance()
   dbr:apply()
 end

-- Determine the number of CPU cores
local cpuCores = tonumber(io.popen("nproc"):read("*n"))


if not cpuCores or cpuCores < 1 then
    cpuCores = 1
end

for i = 1, cpuCores do
    addListeners(i)
    addServers(i)
end

setMaxUDPOutstanding(65535/cpuCores)

setACL("0.0.0.0/0")

My machine has CPU 20C40T x 2

NUMA:                     
  NUMA node(s):           2
  NUMA node0 CPU(s):      0-19,40-59
  NUMA node1 CPU(s):      20-39,60-79

If I set it like this, the performance will be higher

function addListeners(i)
   cn = i-1
    if (cn>=21 and cn<=39) or (cn>=60 and cn<=79) then
        addLocal("10.20.0.49:53", {reusePort=true,cpus={cn}})
    end
end

If I turn on xsk, I have to bind 80 because

ethtool -l ens785f0
Channel parameters for ens785f0:
Pre-set maximums:
RX:             80
TX:             80
Other:          1
Combined:       80
Current hardware settings:
RX:             0
TX:             0
Other:          1
Combined:       80

function addListeners(i)
    local xsk = newXsk({ifName="ens785f0", NIC_queue_id=i-1, frameNums=65536, xskMapPath="/sys/fs/bpf/dnsdist/xskmap"})
    addLocal("10.20.0.49:53", {reusePort=true,cpus={i-1},xskSocket=xsk})
end

So it may be because of the machine. I think I may not be suitable to turn on xsk. Thank you very much.

[rgacogne]

I found a magical place, when kxdpgun qps=200w time=30s, the replies exceeded the requests.

Wow, that's really weird. I'll see if I can reproduce this.

I said the performance is not as expected, I stress tested 100w 200w 300w 500w
If qps is less than or equal to 200w, the performance is better with xsk turned on, otherwise it is higher without turning it on.
If you have time, can you help me see if there is any optimization in my configuration file?

Sure! The first thing that comes to mind is that you might running out of available XSK frames. In my own tests I'm using 1048576 instead of 65536 for the frameNums parameters of newXsk. Note that the value has to be a power of 2.
The second thing is that you might be running out of UDP states in dnsdist itself. The short version is that dnsdist needs to keep some state for all queries that have been sent to a backend, until the corresponding response comes back or a timeout occurs. We can only use the query ID for that, and in DNS it is a 16-bit field, which means we can only have 65536 queries in-flight per backend by default , which is not much, so if the time between the query is sent and the corresponding response (latency) increases at some point, we need more states.
I see that you are setting:

setMaxUDPOutstanding(65535/cpuCores)

which in my humble opinion is a terrible idea since it reduces the number of available states. I would suggest keeping the default (maximum) value.
You can do that get more states per backend by telling dnsdist to use more sockets between dnsdist and the backend, since you then get 65536 states per socket. You can do that by passing the sockets option to newServer. I would suggest using a low value like 64 which should already get rid of the potential states bottleneck.

Finally, I would suggest tuning the OS itself if you have not done yet. A good start would be:

• setUDPSocketBufferSizes(16777216, 16777216) in dnsdist.conf (I see you have it already, never mind!)
• Make sure that the system is not throttling your CPUs: sudo cpupower frequency-set -g performance
• Ensure that incoming packets are correctly distributed amongst cores and RX queues, based on the tuple "src IP, dst IP, src port, dst port": sudo ethtool -N ens785f0 rx-flow-hash udp4 sdfn

Please let me know if it helps, or even if it doesn't! I would like to update the configuration and since you have a quite different hardware than the one I use, I would like to know whether it behaves differently.

[serious-snow]

setMaxUDPOutstanding(65535/cpuCores) --> sockets =64
sudo ethtool -N ens785f0 rx-flow-hash udp4 sdfn

I made the above modifications, and the performance looks similar on my machine.
Maybe because I use leastOutstanding, setting setMaxUDPOutstandingis smaller to make requests more balanced. So the results are not much different. I tried other Load Balancer strategies that were not as high as the default leastOutstanding. My two downstream servers have the same configuration.
I won't test xsk for now, so frameNums = 1048576 will test it next time.
thanks very much.

[serious-snow]

Later I found the cause of the crash, it was a driver problem, I updated the driver and it worked, I tried other distributions and there was no problem.

I probably won't continue to study xdp for dnsdist recently.
Thanks again.

[serious-snow]

ok, thanks, tomorrow do it ---- Replied Message ---- From Remi ***@***.***> Date 06/28/2024 18:51 To ***@***.***> Cc ***@***.***>***@***.***> Subject Re: [PowerDNS/pdns] dnsdist how to use xsk by xdp.py (Discussion #14383) Alright, can you try dumping the dnsdist metrics by executing dumpStats() in the console? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>