[Bug]: Some protocol violations or bugs in VerneMQ

[songxpu]

Environment

• VerneMQ Version: 2.0.0-rc1-5-gf0e6dc15
• OS: Ubuntu 20.04
• Erlang/OTP version (if building from source):
• Cluster size/standalone:

Current Behavior

VerneMQ receives such a request and returns a successful response message or processes and saves it normally

Expected behaviour

In these scenarios, VerneMQ receives such a request with the expectation that it should abort the connection or reject the message

Configuration, logs, error output, etc.

plz set `allow_anonymous = on`, and most settings are default

Code of Conduct

• I agree to follow the VerneMQ's Code of Conduct

[songxpu]

According to the specification of MQTTv5.0:

[MQTT-3.1.3-12]
The User Name MUST be a UTF-8 Encoded String.

Nevertheless, in transmitting such a packet to VerneMQ, it ought to reject the packet yet paradoxically forward a CONNACK message to the client, which intriguingly includes a success code.

echo 101c00044d51545405c2003c03210014000000048f656d6f000470617373 | xxd -p -r | nc 172.17.0.3 1883

[songxpu]

According to the specification of MQTTv5.0:

[MQTT-3.1.3-4]
The ClientId MUST be a UTF-8 encoded string as defined in Section 1.5.3.

Same situation as above.

echo 100f00044d5154540402003c000331328f | xxd -p -r | nc 172.17.0.3 1883

[songxpu]

According to the specification of MQTTv5.0:

[MQTT-3.3.4-6] 
A PUBLISH packet sent from a Client to a Server MUST NOT contain a Subscription Identifier.

However, if we send such a packet including Connect and Publish:

106100044d5154540540b3b037119afb60e317001901215c5326000f766e72366d4541644d78553c44327800173049574d36324268715a6179524b5a62536749534a31360013317675535434755733374e64397846585a38570008676235716d583636
3554000841684b3146454962f71a16010109000c79a506aff5eef39ed5210cd60bba849b523155596942337761334c376e765936573739413862666a46414e4e3172647544345773415778724a6667386d3258653363

x

[songxpu]

According to the specification of MQTTv5.0:

[MQTT-3.1.3-12]
If the User Name Flag is set to 1, the User Name is the next field in the Payload. The User Name MUST be a UTF-8 Encoded String.

Send such a packet (user name contains non-UTF-8 encoded strings):

echo 101c00044d51545405c2003c03210014000000048f656d6f000470617373 | xxd -p -r | nc 172.17.0.3 1883

[songxpu]

According to the specification of MQTTv5.0:

[MQTT-3.3.2-1] The Topic Name MUST be present as the first field in the PUBLISH packet Variable Header. It MUST be a UTF-8 Encoded String.

[MQTT-3.8.3-1] The Topic Filters MUST be a UTF-8 Encoded String.

Send such a Publish packet ([MQTT-3.3.2-1]):

101000044d5154540502003c032100140000
3007000339321a0031

Send such a Subscribe packet ([MQTT-3.8.3-1]):

101000044d5154540502003c032100140000
8209000100000339321a00

The subject of the above two packets (containing illegal UTF-8 encoding) is the same, and thus can constitute a message transfer between two clients.

---

Also

[MQTT-3.10.3-1] The Topic Filters in an UNSUBSCRIBE packet MUST be UTF-8 Encoded Strings.

Send such an Unsubscribe packet:

104800044d5154540502198e1d215415260008744a613168307079000d77516d586452786d3835676132001e32746e6233766c6a304630344d317a6b3855744b45524a506a347579466a
a28f01cb302d260016746e5a725a655353654471496a6a69334c6472686b4100127471745278344c7657457864724b70656e460006645535537a77000445624a490009364b59444d74495670001559777656754e706a53646d654b4f6b4c316e466c430012644b4a69677474757974696c544d70684152000172000339321a00114353696d34303034427951596846436373 

[songxpu]

According to the specification of MQTTv5.0:

It is a Protocol Error to include Authentication Data if there is no Authentication Method.

Send the following packet:

echo 10c10100044d515454052ee8504d1193e14c4a1600154b47574f55355341677341784d654a39437071656921958c22bac526001635795733507a4e773069426875566e626c504e32574e000f43794c714538595549326663643672001d59507459787a4b4d736374486e305050336c384466335848486258304c1d010008000e56466f324b4c6f526c327058346c090007687186e0884020001a5847674b484f58756c6e5847414f6639667866705533646c6951000e78334a795477747a70486b4d3277 | xxd -p -r | nc 172.17.0.3 1883

[songxpu]

According to the specification of MQTTv3.1.1:

SUBSCRIBE, UNSUBSCRIBE, and PUBLISH (in cases where QoS > 0) Control Packets MUST contain a non-zero 16-bit Packet Identifier.

We send the following packet (Connect: Qos 1, Publish: Packet Id 0):

105d00044d515454040c8e3c0018546a37786f3258556551443047644e385950706d51613251001c54374578706f4e686d4679485874384b4c483872554247764935385500196f65486351376f6a694d48596355524d3341533052686b7164
a2110000000d357a386333384e6f416a54305a 

The expectation was that such a request would be denied, but unfortunately it was received and processed.

[ioolkos]

@songxpu Thank you for reporting protocol issues/breaking points to VerneMQ and other brokers! 🥇 👍
We'll review and decide where and how fixes are needed.

---

👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq
👉 Using the binary VerneMQ packages commercially (.deb/.rpm/Docker) requires a paid subscription.

[mths1]

According to the specification of MQTTv5.0:

[MQTT-3.3.4-6] 
A PUBLISH packet sent from a Client to a Server MUST NOT contain a Subscription Identifier.

However, if we send such a packet including Connect and Publish:

106100044d5154540540b3b037119afb60e317001901215c5326000f766e72366d4541644d78553c44327800173049574d36324268715a6179524b5a62536749534a31360013317675535434755733374e64397846585a38570008676235716d583636
3554000841684b3146454962f71a16010109000c79a506aff5eef39ed5210cd60bba849b523155596942337761334c376e765936573739413862666a46414e4e3172647544345773415778724a6667386d3258653363

@ioolkos : This one is interesting. I can reproduce, but it seems that the publish is never executed. I can see valid MQTT in Wireshark...

[ioolkos]

@mths1 yeah, the spec is relatively clear that it's a protocol error. So, I guess we should close the connection.
(Not sure I understand "the publish is never executed" correctly).

---

👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq
👉 Using the binary VerneMQ packages commercially (.deb/.rpm/Docker) requires a paid subscription.

[mths1]

@ioolkos : It means a subscribe on # won't show anything and I set a debug message on the "publish" frame which never seems to be executed.

[songxpu]

@ioolkos : It means a subscribe on # won't show anything and I set a debug message on the "publish" frame which never seems to be executed.

I want to know how you replay the publish packet.
In my tests, I sent client and publish to vernemq separately, rather than including both packets in a message, this is because only the first connect packet can be parsed.

for example:

then I send the following packets respectively:

106100044d5154540540b3b037119afb60e317001901215c5326000f766e72366d4541644d78553c44327800173049574d36324268715a6179524b5a62536749534a31360013317675535434755733374e64397846585a38570008676235716d583636
3354000841684b3146454962f71a16010109000c79a506aff5eef39ed5210cd60bba849b523155596942337761334c376e765936573739413862666a46414e4e3172647544345773415778724a6667386d3258653363

The subscribe client successfully received such publish.

[mths1]

@ioolkos : Regarding the Packet Identifier > 0 problem, would be easy to fix but another check on each publish. I am honestly not sure why the spec does not allow 0 :-) (other than philosophical discussions about if 0 is a number or not)

[mths1]

@songxpu : Yeah, I should have been more detailed here. When I do two requests it works as expected (or not expected :-) ). I was not sure what the correct behaviour is in the "all at once" case.

[mths1]

Hello @ioolkos : All mentioned issues should be addressed by the pull requests. Comments are welcome, otherwise I will move from draft to final :-)

[ioolkos]

@songxpu : Yeah, I should have been more detailed here. When I do two requests it works as expected (or not expected :-) ). I was not sure what the correct behaviour is in the "all at once" case.

I think it's related to the fact that the tests here don't use an MQTT client state machine. Clients wait to get a CONNACK from the server before they send messages.

Although.... technically clients are allowed to send PUBLISH's to the server right away, that is: before getting a CONNACK (at their own risk because the messages will be lost if the client gets rejected). I need to verify whether Verne accepts PUBLISHs at all before sending out a CONNACK and have the MQTT session fully in connected state but I don't think Verne accepts this. Note that the problem with supporting that is that the server has to "hold back" processing the PUBLISHs until it has confirmed he connection; otherwise it would have processed messages from an unauthorized client.

---

👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq
👉 Using the binary VerneMQ packages commercially (.deb/.rpm/Docker) requires a paid subscription.

[mths1]

@songxpu : Thanks for the report. All issues should be addressed by the linked pull requests. Feel free to test them if you want. May I ask what is the background of you performing all those tests? (feel free to do more, though :-) )

[songxpu]

@songxpu : Thanks for the report. All issues should be addressed by the linked pull requests. Feel free to test them if you want. May I ask what is the background of you performing all those tests? (feel free to do more, though :-) )

Hi @mths1, I performed those tests for the purpose of course work.

By the way, I would like to ask Vernemq how to clear or reset persistent data?
For example, Vernemq currently saves many session and topic messages, and I hope to delete them so that after restarting Vernemq, testing can start in a brand new environment.

[ioolkos]

@songxpu to fully remove any state on a VerneMQ node (for example between test runs), you can mv away or delete everything that's under the VerneMQ data directory. VerneMQ will then recreate all the needed directories upon boot; all state will be fresh (no subscriptions, messages, cluster state etc.)

---

👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq
👉 Using the binary VerneMQ packages commercially (.deb/.rpm/Docker) requires a paid subscription.