Welcome to the exciting world of humanoid robotics! In this article, we will guide you through the process of implementing the Humanoid Shadowing Transformer (HST) and the Humanoid Imitation Transformer (HIT). We’ll also show you how to get started with whole-body pose estimation.

Getting Started with Humanoid Shadowing Transformer (HST)

The HST uses reinforcement learning and is built upon two robust frameworks: legged_gym and rsl_rl.

Installation Steps

• First, install IsaacGym v4 from the official source.
• Place the isaacgym folder inside the HST directory.
• Open your terminal and run the following commands to set up the environment:

cd HST
cd rsl_rl
pip install -e .
cd ../legged_gym
pip install -e .

Example Usages

To train HST, simply run the following command:

python legged_gym/scripts/train.py --run_name 0001_test --headless --sim_device cuda:0 --rl_device cuda:0

To use a trained policy, you can execute:

python legged_gym/scripts/play.py --run_name 0001_test --checkpoint -1 --headless --sim_device cuda:0 --rl_device cuda:0

Understanding the Code with an Analogy

Imagine you’re teaching a child to ride a bike. You start with a balance bike, allowing them to grasp the idea of balance without worrying about pedaling. This is akin to how the HST is trained in a simulation environment—allowing the model to understand actions (like riding) before trying it in a real-world setting. Once they’ve mastered balancing, you introduce pedals (real-world applications), gradually helping them transition from simulation to reality with confidence!

Implementing Humanoid Imitation Transformer (HIT)

The HIT focuses on imitation learning in a real-world scenario, relying on the ACT repository and Mobile ALOHA repository.

Installation Steps

• First, create a conda environment:

conda create -n HIT python=3.8.10

• Activate the environment:

conda activate HIT

• Then install the necessary packages:

pip install torchvision torch pyquaternion pyyaml rospkg pexpect mujoco==2.3.7 dm_control==1.0.14 opencv-python matplotlib einops packaging h5py ipython getkey wandb chardet h5py_cache

• Lastly, navigate to the HIT directory and install the detr module:

cd HIT/detr
pip install -e .

Example Usages

You can collect your own data or download a dataset from here. Make sure to place it in the HIT folder.

To train the HIT for a specific task, run:

python imitate_episodes_h1_train.py --task_name data_fold_clothes --ckpt_dir fold_clothes --policy_class HIT --chunk_size 50 --hidden_dim 512 --batch_size 48 --dim_feedforward 512 --lr 1e-5 --seed 0 --num_steps 100000 --eval_every 100000 --validate_every 1000 --save_every 10000 --no_encoder --backbone resnet18 --same_backbones --use_pos_embd_image 1 --use_pos_embd_action 1 --dec_layers 6 --gpu_id 0 --feature_loss_weight 0.005 --use_mask --data_aug --wandb

Setting Up the Hardware Codebase

The hardware codebase utilizes unitree_ros2.

Installation Steps

• Install the unitree SDK from here.
• Now install the unitree ROS2 package from here.
• Create a low-level conda environment:

conda create -n lowlevel python=3.8

• Activate the environment:

conda activate lowlevel

• Install NVIDIA Jetpack:

install [nvidia-jetpack](https://docs.nvidia.com/jetson/archives/jetpack-architecture/jetpack-4.6/install-jetpack/index.html)

• Finally, install PyTorch and torchvision by following the links provided in the README for the specific versions needed.

Example Usages

To deploy your trained policy, put it in the hardware-script/ckpt folder, renaming it to policy.pt. Then run:

python hardware_whole_body.py --task_name stand

Pose Estimation

For body pose estimation, take a look at WHAM, and for hand pose estimation, refer to HaMeR.

Troubleshooting

If you encounter issues during the setup or execution process, consider the following troubleshooting tips:

• Ensure that all dependencies are correctly installed; missing packages can lead to errors.
• Check your environment configuration—different tasks may require specific GPU configurations.
• If the code doesn’t execute as expected, verify that you are in the correct directory and that all folder paths are accurate.
• For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.