If you’re aiming to imbue robots with the ability to perform complex tasks over extended periods, you might be interested in the Sequential Dexterity project. This blog post guides you through its installation, setup, training, and evaluation, allowing you to explore the wonders of robotic dexterity.

Table of Content

• Overview
• Installation
• Quick Demo
• Training
• Evaluation
• Acknowledgement
• Citations
• License

Overview

This repository presents the implementation of the paper Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulation by Yuanpei Chen, Chen Wang, Li Fei-Fei, and C. Karen Liu at The Movement Lab and Stanford Vision and Learning Lab. It consists of complete implementation code, along with demos, training procedures, evaluation techniques, and a real-world system.

Installation

Setting up the environment is straightforward. Follow these steps:

• First, create a new Python environment:

conda create -n seqdex python=3.8

conda activate seqdex

• Next, install IsaacGym (test it with Preview Release 34 or 44). Follow the instructions here to download the package:

tar -xvf IsaacGym_Preview_4_Package.tar.gz
cd isaacgym
python pip install -e .
cd examples
# Test installation
python examples/joint_monkey.py

• Clone the SeqDex repository:

git clone https://github.com/sequential-dexterity/SeqDex.git
cd SeqDex
pip install -r requirements.txt
pip install -e .

• Lastly, download the trained checkpoint from here:

unzip seqdex_checkpoint.zip
mv checkpoint SeqDex/dexteroushand/envs/checkpoint

Quick Demo

To experience a quick demo of the project, you can assign LEGO building manuals interactively to the robot. Start the demo by following these commands:

cd SeqDex/dexteroushand/envs
python train_rlgames.py --task BlockAssemblyGUI --num_envs=1 --play

The control instructions for the interaction are provided in the demo interface:

Training

Training in this project is akin to teaching a child to complete a LEGO set. Each sub-task is like a different step in the manual, leading toward the final assembly.

To train the Block Assembly task, you’ll execute the following commands:

python scripts/bi-optimization.py --task=[BlockAssembly ToolPositioning]

Training for each sub-task takes approximately 1–2 days, so be patient. You can train sub-tasks individually:

python train_rlgames.py --task=[BlockAssemblySearch BlockAssemblyOrient BlockAssemblyGraspSim BlockAssemblyInsertSim] --num_envs=1024

The trained model is saved to the runs folder, while the terminal state will be stored within immediate_state. These states can further train the transition feasibility function:

python policy_sequencing_tvalue_trainer.py --task=[BlockAssemblySearch BlockAssemblyOrient BlockAssemblyGraspSim BlockAssemblyInsertSim]

Evaluation

To evaluate a trained model without initiating a new training process, provide the following command:

python train_rlgames.py --task=BlockAssemblyGraspSim --checkpoint=.checkpoint/block_assembly/last_AllegroHandLegoTestPAISim_ep_19000_rew_1530.9819.pth --play --num_envs=256

The evaluation will calculate the overall success rate across sub-tasks, aiding in validating the trained policies:

python scripts/evaluation.py

You can also conduct a LEGO assembly test using the demo program:

cd SeqDex/dexteroushand
python train_rlgames.py --task BlockAssemblyGUI --num_envs=1 --play --record_video --record_lego_type=0

The results will be saved in output_video.

Real-world Deployment

This section shares the implementation code of the robot system used in real-world experiments, including:

• Allegro hand controller (zmq)
• Visual tracking system (redis)

The franka panda arm controller is robustly managed using Deoxys, allowing for modularized robotic control.

Acknowledgement

We thank the contributors from Bi-DexHands for their support and contributions to this project.

Citations

Please cite the work if you use this repository in your publications:

@article{chen2023sequential,
  title={Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulation},
  author={Chen, Yuanpei and Wang, Chen and Fei-Fei, Li and Liu, C Karen},
  journal={arXiv preprint arXiv:2309.00987},
  year={2023}
}

License

This project is licensed under the MIT License.

Troubleshooting

If you encounter issues during installation or execution, consider the following troubleshooting ideas:

• Ensure your Python version matches the requirements (Python 3.8).
• Verify that all packages in requirements.txt are installed successfully.
• Check your internet connection if downloads are failing.
• Double-check your command syntax in the terminal.

For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.