How to update the dat.gui panel programmatically

[diegoferigo]

Hi @rdeits, thanks for all your work with meshcat!

I'm trying to obtain using Python something similar to what was done in drake for controlling the displayed panel programmatically, i.e. adding (at least) buttons with callbacks and (possibly) sliders.

As inexperienced web developer (read, never developed any web-related application), here below I summarize what I learned from the repos:

• The communication is layered in two steps:
  1. meshcat.Visualizer ⇐ zmq ⇒ zmq server
  2. zmq server ⇐ WebSockets ⇒ Browser
• The displayed control panel in the Browser uses dataarts/dat.gui, defined in index.js#L297
• Recently, PRs improved the way to interact with the control panel:
  • rdeits/meshcat#94
  • rdeits/meshcat#98
  • rdeits/meshcat#100
  • rdeits/meshcat#129)
• It seems that it's possible to use the zmq server to interact with the controls RussTedrake/manipulation#145

To conclude this summary, drake eventually dropped the zmq additional step and started communicating directly with the WebSocket (RobotLocomotion/drake#13038 (comment)). I also found a similar issue you commented few years ago (meshcat-dev/meshcat#44 (comment)) in which you wrote that this is also the approach used in Julia.

This being said, from Python I'm currently missing the connection step between the meshcat.Visualizer (or any inheriting class) and the controls:

1. Let's assume I can start a custom version of meshcat.servers.zmqserver.ZMQWebSocketBridge and define new commands. Do you have any pointers on how to pass through the zmqserver for interfacing with the controls?
2. Instead, by creating together with the Visualizer an additional direct connection to the websocket, what would be the approach to access the control panel? I assume that both connections (this new one + the exising one) could co-exist, but I might be wrong.

---

For future reference, in rdeits/meshcat@test there are static html with experiments that can be opened as follows:

cd meshcat/test
python3 -m http.server
# Browse: http://localhost:8000/<filename>.html

[kielnino]

Hi @diegoferigo, if you are still interested, here is a first idea how to approach this:

In the meshcat-javascript-viewer the set_control-comand is already available (see here). So what is left is to define a command in the commands.py like

class SetControl:
    __slots__ = ["name", "callback", "value", "min", "max", "step", "keycode1", "keycode2"]
    def __init__(self, name, callback, value, min, max, step, keycode1, keycode2):
        self.name = name
        self.callback = callback
        self.value = value
        self.min = min
        self.max = max
        self.step = step
        self.keycode1 = keycode1
        self.keycode2 = keycode2

    def lower(self):
        return {
            u"type": u"set_control",
            u"name": self.name.lower(),
            u"callback": self.callback,
            u"value": self.value,
            u"min": self.min,
            u"max": self.max,
            u"step": self.step,
            u"keycode1": self.keycode1,
            u"keycode2": self.keycode2
        }

and in the zmqserver.py we need to forward the comand by adding a elif case in the handle_zmq function:

def handle_zmq(self, frames):
    [...]
    elif cmd == "set_control":
        self.forward_to_websockets(frames)
        self.zmq_socket.send(b"ok")
    [...]

Now you can send the comand from your python script like:

import zmq
from meshcat.commands import SetControl

context = zmq.Context()
zmq_socket = context.socket(zmq.REQ)
zmq_socket.connect(zmq_url)


def set_control(self):
        cmd_data = SetControl(name='MyControl', callback=None, value='4', min=0, max=10, step=1, keycode1=None, keycode2=None).lower()
        zmq_socket.send_multipart([
            cmd_data["type"].encode("utf-8"),
            "".encode("utf-8"),
            umsgpack.packb(cmd_data)
        ])
        return zmq_socket.recv()