WIP: _meshfile_object loaders.

[danzimmerman]

I've made some potential progress toward #92 and possibly toward #27 and I wanted to open a PR for discussion.

What I've done so far:

• Defined a meshcat.geometry.MeshFileObject class that lowers (is this the right term?) to a _meshfile_object
• Defined a meshcat.geometry.DaeMeshFileObject that inherits from MeshFileObject that for now just has a from_file() static method that uses xml.etree.ElementTree to parse a Collada file for image textures, load the corresponding images (assuming the same directory) and base64 encode them as image resources.
• Modified meshcat.commands.SetObject to deal with MeshFileObjects
• Added an example color_texture_collada_example.ipynb and added required assets to examples (so I don't have to do a PR on meshcat as well). This example pulls UR3 mesh data from Gepetto/example-robot-data for simply visualizing Collada files with colors.

Still a lot of work to do here, OBJ/MTL parsing to add toward #27, and getting some more assets and test cases together, but I could use some feedback on whether I'm on the right track in terms of fitting into the existing code and object hierarchy.

[danzimmerman]

Enabled png or jpeg textures for Collada files based on file extension (switching the base64-encoded image mime type accordingly).

Added Collada assets of the Valkyrie head with both jpeg and png textures with different colors, as well as one in .obj/.mtl format, which will be visualized with red accents.

Updated the notebook to visualize the original head from src/meshcat/viewer/data as well as the jpeg and png textured Collada files. Will add the .obj/.mtl once I add code to parse that.

[rdeits]

Yeah, this suggests that the type hierarchy isn't quite right. Over in the Julia version of meshcat (which is a little more developed), I have AbstractObject as a parent type of Object and MeshFileObject, and we might want something similar here.

https://github.com/rdeits/MeshCat.jl/blob/6ca2e706d2ccb9501a0fb5ccd5829c208f26af4f/src/mesh_files.jl#L53

https://github.com/rdeits/MeshCat.jl/blob/6ca2e706d2ccb9501a0fb5ccd5829c208f26af4f/src/objects.jl#L6

[duburcqa]

Any progress ? I would love to see this feature merged 😃

[danzimmerman]

@duburcqa Sorry, I haven't really had any time to work on this beyond what I've done here.

I had an first prototype where I defined an external class that prepared an object dictionary all in one go and returned it from .lower()... a class instance can be sent to a Meshcat Visualizer viz using viz.window.send() to load a textured or colored Collada file.

https://gist.github.com/danzimmerman/a392f8eadcf1166eb5bd80e3922dbdc5

Might be a temporary workaround?

Hope I can find some more time to work on this soon.

[duburcqa]

@danzimmerman nice, thank you ! My meshcat is already so heavily modded that I'm fine with your proposal :) I will try it asap

[duburcqa]

OK so it is working, each texture image is loaded systematically even if it is used by multiple files. It is slowing done the mesh loading dramatically :/

[danzimmerman]

OK so it is working, each texture image is loaded systematically even if it is used by multiple files. It is slowing done the mesh loading dramatically :/

Is this for a use case like https://github.com/duburcqa/jiminy/tree/master/data/bipedal_robots/atlas/ with files like extremities_diffuse.png?

I was developing against really simple lightweight assets, maybe that's a better test case.

To really deal with that kind of thing it seems we'd need a class that knows about the whole URDF or directory structure and probably also good to figure out if image resources can be cached (like #114 ). I suspect they can 🤔

[duburcqa]

Yes exactly. The issue is that the current mechanism based on filling the optional field "resources" just overwrites the original path/url with another one. I'm not sure it is flexible enough for caching, because it should be a kind of reference to an already loaded texture instead.

[duburcqa]

After checking, it is possible to activate the cache of ThreeJS doing THREE.Cache.enabled = true; and rely on it to avoid sending the same texture repeatedly. It requires to also keep track on python side of the set of textures in order to put them into cache only once.

[danzimmerman]

I'm not sure it is flexible enough for caching, because it should be a kind of reference to an already loaded texture instead.

Yeah, I don't really have a good grip on the data flow after passing the resources yet, or alternatives.

I guess how it's supposed to work is that resources would normally be actual URLs accessible on the network, and the base64 URIs are a workaround to that.

From #27 (comment)

The hardest part of this actually getting the texture file into the right format. I'm using the native object loader in Three.js, which expects that the texture files will be at some URL that it can access.

[duburcqa]

Yes kind of. I implemented a POC using ThreeJS cache to avoid sending and loading the data repeatedly:

var preloaded_resources = {};
function loadImageAsync(key, url) {
    if(preloaded_resources[key] == undefined) {
        preloaded_resources[key] = new Promise((resolve, reject) => {
            let img = new Image();
            img.onload = () => {
                MeshCat.THREE.Cache.add(key, img);
                resolve();
            }
            img.onerror = reject
            img.src = url;
        })
    }
    return preloaded_resources[key];
}

var handle_command = viewer.handle_command;
viewer.handle_command = function(cmd) {
    if (cmd.type == "set_object") {
        (async () => {
            // Store/Load resources in cache
            let resources = cmd.object.object.resources;
            if (resources !== undefined) {
                let promiseTab = [];
                for (const [key, url] of Object.entries(resources)) {
                    promiseTab.push(loadImageAsync(key, url));
                }
                await Promise.all(promiseTab);
                cmd.object.object.resources = {};
            }
            // Handle command now that everything is in cache
            handle_command.call(this, cmd);
        })();
    } else {
        handle_command.call(this, cmd);
    }
};

On Python side, the texture image converted to base64 is only sent the first time, later on it is just an empty string. Anyway it is going to be loaded from the cache. It works because the commands are queued and processed sequentially by design.