Slow performance between Publish/Inject and seeing TX to network

[grantd-singular]

I've been successfully using MQTTnet / Extensions.RPC for quite some time, but recently noticed some performance issues where TX packets sometimes take a long time (1-4 seconds) to be sent after calling Publish.

Originally I was using sepearte instance of MqttServer and MqttClient, and utilziing loopback for communication between client and server, but in troubleshooting I changed to using the Inject and Intercept functionality directly on MqttServer.

System Specs:
.NET6.0
Windows 10
i9 processor
Wired Ethernet connection (dedicated for mqtt traffic. PTP to a target instrument).

I have logs that look like:

2022-10-02 14:53:44.5188 [22] [FC:3] [Lane:] [INFO] Injecting publish for topic Cmd/59012158-3efe-4ed8-b01c-635ac720c7d6/Fluidics/SetPcbValveState: Fluidics/SetPcbValveState message: MQTTnet.MqttApplicationMessage  [MqttLib.MqttRpcService]
2022-10-02 14:53:44.5188 [22] [FC:3] [Lane:] [DEBUG] Client 'cabinet_controller': Queued PUBLISH packet with topic 'Cmd/59012158-3efe-4ed8-b01c-635ac720c7d6/Fluidics/SetPcbValveState'.  [MqttClientSessionsManager]
2022-10-02 14:53:44.5188 [22] [FC:3] [Lane:] [INFO] Injecting publish sent for topic Cmd/59012158-3efe-4ed8-b01c-635ac720c7d6/Fluidics/SetPcbValveState: Fluidics/SetPcbValveState  [MqttLib.MqttRpcService]


... (Irrelevant lines removed, but it's mostly RX of ~100 bytes every 100 ms or so .   These are lower priority messages for me. I made sure to publish any events from these messages being received on independent tasks so not to hold up MqttChannelAdapter from further processing )


2022-10-02 14:53:48.5169 [71] [FC:] [Lane:] [DEBUG] TX (74 bytes) >>> Publish: [Topic=Cmd/59012158-3efe-4ed8-b01c-635ac720c7d6/Fluidics/SetPcbValveState] [Payload.Length=4] [QoSLevel=AtMostOnce] [Dup=False] [Retain=False] [PacketIdentifier=0]  [MqttChannelAdapter]

Any suggestion? I feel like Task and await is contributing, and am considering porting to dedicated standard Threads that can be more easily prioritized