Welcome to this guide on implementing the Topological Semantic Graph Memory (TSGM) for image goal navigation tasks. In this article, we’ll walk you through the setup, data generation, training processes, and visualization of your results. Think of TSGM as a smart map that helps a robot navigate through a new environment by intelligently remembering and organizing the visuals it observes, leading it efficiently to its destination.

Understanding TSGM

Imagine you are a tourist in a new city. Instead of relying purely on a physical map that shows you the streets, you start creating a mental map every time you notice landmarks—restaurants, parks, or historical buildings. Similarly, TSGM helps a robot by combining the details of various visual objects (like landmarks) into a graph memory that it can refer back to while navigating. Its three components include a Graph Builder, a Cross Graph Mixer, and a Memory Decoder, all working together to create this mental map.

Installation Guide

To get started with TSGM, follow these installation steps:

• Ensure you have the following software installed:
  • Python 3.7
  • Pytorch 1.10
  • Detectron2
  • Habitat-sim (version 0.2.1)
  • Habitat (version 0.2.1)
• Create a new Anaconda environment:

  conda env create -f environment.yml
  conda activate tsgm

• Install necessary libraries:

  conda install habitat-sim==0.2.1 withbullet headless -c conda-forge -c aihabitat
  cd mkdir programs
  cd programs
  git clone --branch stable https://github.com/facebookresearch/habitat-lab.git habitat-lab-v21
  cd habitat-lab-v21
  git checkout tags/v0.2.1
  pip install -e .
  conda activate tsgm
  conda install pytorch==1.10.0 torchvision==0.11.0 torchaudio==0.10.0 cudatoolkit=11.3 -c pytorch -c conda-forge
  python -m pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu113/torch1.10/index.html

Setting Up the Environment

Most scripts assume that the Gibson dataset is located in the habitat-lab/data folder. You can structure your habitat-lab directory as follows:

habitat-lab
└── data
    └── datasets
        └── pointnav
            └── gibson
                └── v1
                    ├── train
                    └── val
            └── scene_datasets
                └── gibson
                    ├── *.glb, *.navmesh

Data Download and Generation

You can download the dataset [here](https://drive.google.com/drive/folders/1-8srO6kwxf_qxnIWdH6K7AToP_oxWEYg?usp=sharing). Place the downloaded data in the TopologicalSemanticGraphMemory/data directory.

Next up, let’s generate the data:

• For Imitation Learning:

  python collect_il_data.py --ep-per-env 200 --num-procs 4 --split train --data-dir IL_data/gibson

• For Topological Semantic Graph:

  python collect_graph.py ./configs/TSGM.yaml --data-dir IL_data/gibson --record-dir IL_data/gibson_graph --split train --num-procs 16

Training Your Model

Once the data is prepared, you can proceed with training your model:

• Imitation Learning:

  python train_il.py --policy TSGMPolicy --config ./configs/TSGM.yaml --version exp_name --data-dir IL_data/gibson --prebuild-path IL_data/gibson_graph

• Reinforcement Learning:

  python train_rl.py --policy TSGMPolicy --config ./configs/TSGM.yaml --version exp_name --diff hard --use-detector --strict-stop --task imggoalnav --gpu 0,1

Evaluation and Visualization

After training, we can evaluate the results:

python evaluate.py --config ./configs/TSGM.yaml --version version_name --diff hard --gpu 0

To visualize the results, you can run:

python visualize_tsgm.py --config-file ./configs/TSGM.yaml --eval-ckpt checkpoint_path

Troubleshooting

Should you encounter issues during the installation or execution, consider the following troubleshooting steps:

• Double-check your Python and package versions as mismatches can often lead to unforeseen errors.
• Ensure that the paths to datasets and models are correctly set in the configuration files.
• If you have problems with data generation, verify that you are using the correct directory structure as specified.
• Errors during training usually stem from insufficient GPU memory; consider reducing batch size or using a less complex model.

For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

Conclusion

At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.