Help with implementation and grabbing logic of the shadow hand in mujoco + guidance of how to implement reinforcment learning too

[brunohawkins]

Intro

Hi!

I am an undergrad studentat UCL, I use MuJoCo for my research on bionic hands.

My setup

pycharm api, latest version of mujoco. powerful windows pc with 16 gb ram and rtx3070

My question

Long story short, my goal is to use reinforcement learning and computer vision to augment robotic prosthetics for a more intuitive, user-friendly experience when picking up household objects. I have decided to do this in a mujoco and use the shadow hand to complete the project.

I want to compare the performance of 3 different models:

Pre-programming the shadow hand without any AI using logic to grab, pick up, place back down and release a 3D model of a cup.

Use reinforcement learning so the hand can 'learn' how to pick up the cup optimally (e.g. Using enough gripping strength to pick up the cup, minimising energy consumption of the shadow hand)

To see if extracting image data about the cup can further improve gripping, use a combination of object detection and reinforcement learning from model 2.

I have already trained an object detection model and have started understanding how mujoco and XML file formats work. I already have a background in python, so I am using the official mujoco python bindings to help with this project.

I am struggling to implement the first two models and was wondering if anyone has any guidance.

I have focused since the start of January on model 1, however, getting the logic right to do this has proven to be a challenge in itself. I am essentially treating the shadow hand as a free joint and getting it to 'float' towards the cup.

However, this isn't working very well and wondered if you had a different suggestion to approach this.

My background in robotics is limited, but if it is easier to attach the shadow hand to a robotic arm in the simulator, do you recommend I do this?

Secondly, I want to get started on model 2 as soon as I have implemented model 1 (as I assume there may be some crossover between the two in terms of the python code).

I would like to know the support for reinforcement learning surrounding gripping objects with the shadow hand and how to implement this.

The key metrics I am looking to analyse is:
Model training time
Gripping strength in each model based on force sensors
How long each gripping cycle takes to complete
Processing power and energy consumption

I appreciate this is a lot of information to take in and that this is quite a specific request, but any help or guidance would be greatly appreciated. I'm focusing on the first 2 models for now cause the cv part will be its own challenge.

Minimal model and/or code that explain my question

Using shadow hand model here, edited orientation so it starts in a way that the arm would be angled to pick up a cup: https://github.com/google-deepmind/mujoco_menagerie/blob/main/shadow_hand/README.md

Implemented in a scene with a mug object here:

<mujoco model="shadow_hand_and_mug">
  <!-- Include the Shadow Hand -->
  <include file="C:\Users\bruno\PycharmProjects\FinalYearResearchProjectUCL\Project_Files\robotics_stuff\shadow_hand\right_hand_rotated.xml" />

  <statistic extent="0.3" center="0.3 0 0" />

  <visual>
    <rgba haze="0.15 0.25 0.35 1" />
    <quality shadowsize="8192" />
    <global azimuth="220" elevation="-30" />
  </visual>

  <asset>
    <texture type="skybox" builtin="gradient" rgb1="0.3 0.5 0.7" rgb2="0 0 0" width="512" height="3072" />
    <texture type="2d" name="groundplane" builtin="checker" mark="edge" rgb1="0.2 0.3 0.4" rgb2="0.1 0.2 0.3"
      markrgb="0.8 0.8 0.8" width="300" height="300" />
    <material name="groundplane" texture="groundplane" texuniform="true" texrepeat="5 5" reflectance="0.2" />
    <mesh name="test_cup" file="C:/Users/bruno/PycharmProjects/FinalYearResearchProjectUCL/Project_Files/robotics_stuff/Testing_mujoco_environment/test_cup.obj" scale="0.0005 0.0005 0.0005" />
    <material name="plastic" rgba="0.8 0.8 0.8 1" />
  </asset>

  <worldbody>
    <!-- Lights -->
    <light pos="0 0 1" />
    <light pos="0.3 0 1.5" dir="0 0 -1" directional="true" />

    <!-- Floor -->
    <geom name="floor" pos="0 0 -0.1" size="0 0 0.05" type="plane" material="groundplane" />

    <!-- Cup -->
    <body name="cup" pos="1 0 0" quat="0.707 0.707 0.707 0.707">
      <geom type="mesh" mesh="test_cup" material="plastic" mass="0.24" />
      <joint name="cup_free_joint" type="free" />
    </body>
  </worldbody>
</mujoco>

python script that has the shadow hand and mug in a scene, and the hand is trying to do a gripping motion, but doesn't work. no reinforcement learning is implemented yet:

import mujoco
import mujoco_viewer
import xml.etree.ElementTree as ET
import numpy as np

# Parse the keyframes.xml file
def parse_keyframes(file_path):
    """
    Parses the keyframes XML file and returns a dictionary of keyframe names and their corresponding qpos.
    
    Args:
        file_path: The path to the keyframes.xml file.

    Returns:
        A dictionary where keys are keyframe names and values are lists of qpos (joint positions).
    """
    # Parse the XML tree
    tree = ET.parse(file_path)
    root = tree.getroot()
    keyframes = {}
    # Iterate through each keyframe in the XML and extract its name and qpos values
    for key in root.find('keyframe'):
        name = key.attrib['name']  # Keyframe name
        qpos = list(map(float, key.attrib['qpos'].split()))  # Joint positions as a list of floats
        keyframes[name] = qpos
    return keyframes

# Apply PD control to stabilize joints
def apply_joint_control(model, data, target_qpos, kp=3000, kd=150):
    """
    Applies proportional-derivative (PD) control to stabilize the hand joints against gravity.

    Args:
        model: The MuJoCo model object.
        data: The MuJoCo data object, containing the current state of the simulation.
        target_qpos: The target joint positions for the shadow hand.
        kp: Proportional gain for PD control (affects position correction).
        kd: Derivative gain for PD control (affects velocity damping).
    """
    for i, target in enumerate(target_qpos):
        # Ensure we do not exceed the number of actuators in the model
        if i >= model.nu:
            break  # Exit loop if the index exceeds the number of controls (nu)
        # Compute position error (difference between target and current position)
        position_error = target - data.qpos[i + 7]  # +7 skips global degrees of freedom (DOFs)
        # Compute velocity error (negation of current joint velocity)
        velocity_error = -data.qvel[i + 7]
        # Compute the control torque using PD control
        control = kp * position_error + kd * velocity_error
        # Apply the computed torque to the actuator
        data.ctrl[i] = control

# Main function
def main():
    """
    Main function to run the MuJoCo simulation for the shadow hand gripping task.
    """
    # File paths for the scene XML and keyframes XML
    scene_path = r"C:\Users\bruno\PycharmProjects\FinalYearResearchProjectUCL\Project_Files\robotics_stuff\shadow_hand\scene_shadow_hand_and_mug.xml"
    keyframe_path = r"C:\Users\bruno\PycharmProjects\FinalYearResearchProjectUCL\Project_Files\robotics_stuff\shadow_hand\keyframes.xml"

    # Load the MuJoCo model (scene configuration) and data (simulation state)
    model = mujoco.MjModel.from_xml_path(scene_path)
    data = mujoco.MjData(model)

    # Parse the keyframes from the keyframes.xml file
    keyframes = parse_keyframes(keyframe_path)

    # Extract the "pre grasp" configuration from the keyframes
    pre_grasp_qpos = np.array(keyframes["pre grasp"])

    # Initialize the MuJoCo viewer for visualizing the simulation
    viewer = mujoco_viewer.MujocoViewer(model, data)

    # Simulation loop
    while viewer.is_alive:
        # Apply PD control to maintain the hand in the "pre grasp" position
        apply_joint_control(model, data, pre_grasp_qpos, kp=3000, kd=150)

        # Step the simulation forward
        mujoco.mj_step(model, data)
        # Render the simulation in the viewer
        viewer.render()

    # Close the viewer after the simulation loop ends
    viewer.close()


if __name__ == "__main__":
    main()

Confirmations

• I searched the latest documentation thoroughly before posting.
• I searched previous Issues and Discussions, I am certain this has not been raised before.