Messaging Improvement

[Mossaka]

Had a hour-long chat with @danbugs to discuss areas of improvments for the messaging implementaitons, and we have identified a few action items that can immediate start working on to give a much better messaging experience.

Start a messaging client per thread

Currently all the messaging clients are included in a locked hashmap which limits the concurrency to only one client that can receive messages. This is obviously not going to be able to scale out. The solution to fix this is to create a client in its own thread in the host and the client thread will publish/receieve messages for that specific client.

open function will create a new client

Current implementation creates one client per subscription of a topic. This multiplies the number of connecitons for each running thread/process because if there are 10 consumers and each consumer subscribes to 10 topics, the number of clients being created are 10 * 10 = 100. Ideally, the number of clients should be equal to the number of consumers + producers. Thus, a client can subscribe to multiple topics.

To make that above happen, we will need to create a client in the level of open instead of subscribe.

Wildcard in resource namespaces

We should allow wildcard or "star" namespaces in slightfile so that users can declare as many consumer/producer as they want in the configuration file. This work needs a seperate issue and possibly a proposal to flesh out the details of the what it means to have star namespace for other capabilities like keyvalue. But from an initial discussion, we thought it would be useful for other capabiltiies to use a wildcard namespace as well.

Support competing consumers a.k.a queues

This is the MQ capability we had before but with the new messaging capability that merges the mq and pub/sub capabiltieis, we introduced a regression that we no longer have competing consumers. Further discussion and design proposal are probably needed to layout if the channel type could be helpful in distinguishing message queues and topics in slightfile.

I will a sub-task for each of the action items described above.

• Messaging: Start a messaging client per thread #349
• Messaging: open function will create a new client #350
• Slightfile: Wildcard in resource namespaces #351
• Messaging: Support competing consumers a.k.a queues #352