FIFO/LIFO message queues

[jules-ch]

It seems to me the message queue for MQTT is using a FIFO queue.
I'm using owntracks to stream my position and displays it on a web application.

It seems to me the last positions should be sent in priority to have the last position first.
Is it somthing that can be considered as configuration ?

When the applicaiton is offline for quite a while, offline messages is sent in the order they were created.

[growse]

When the applicaiton is offline for quite a while, offline messages is sent in the order they were created.

This is by design - both because it's more efficient for the remote endpoint's database to receive and insert locations in timestamp order, and because the time taken to dump a large number of locations onto MQTT after a period offline is not large (MQTT will buffer / deliver them as appropriate without blocking the client).

I'm interested to understand more the specific problem / use-case where that doesn't work though. Are you finding it takes too long to catch-up after a period offline?

[jules-ch]

Thanks for the swift response.

Well I'm using this app to stream position to a MQTT broker that store the position in a Time series DB to be processed in real time.
My use case is a biking trip real time tracker. It worked really well, but when I'm getting offline with my mobile the position is streamed with FIFO from the buffer.
The frequency of update for real time position was like every 2s. When positions are pushed to the broker after several hours offline, past position have priority, and I can see my position being streamed on the web front end and it can take several minutes. (between 1 and 5)

I'm not saying anything bad about the implementation, just wanted to know why last position was not the priority. :)
It's a minor convenience.

[growse]

So I can probably make sure the inner queue is actually a deque, which will let the app poll from either end. I might put it behind an experimental feature flag for the next release, as I'm not sure what else might break.

Does that work?

[jules-ch]

That would work for me, if that's not too much work on your end. :)

[jules-ch]

I'll try to ake some benchmark on different buffersize with offline status.

[jules-ch]

Ok did some benchmarks.
One of the problems I was hitting is the publish rate throttling of some cloud platforms.
Like AWS IoT Core that throttles to 100 messages/s per connections, this limits the mobile application when publishing back after being offline.

I'm getting better results with a dedicated mosquitto instance:
500 positions -> 3s to be processed and written in DB