CVPR 2024

Nikhil Keetha · Jay Karhade · Krishna Murthy Jatavallabhula · Gengshan Yang · Sebastian Scherer · Deva Ramanan · Jonathon Luiten

Paper · Video · Project Page

Table of Contents

Table of Contents

1. Installation
2. Online Demo
3. Usage
4. Downloads
5. Benchmarking
6. Acknowledgement
7. Citation
8. Developers

Installation

To set up SplaTAM, follow these steps to install the required dependencies:

It is recommended to use:

• Python 3.10
• Torch 1.12.1 & CUDA=11.6

The simplest method is to use Anaconda and pip:

conda create -n splatam python=3.10
conda activate splatam
conda install -c nvidia label cuda-11.6.0 cuda-toolkit
conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.6 -c pytorch -c conda-forge
pip install -r requirements.txt

Alternatively, utilize the conda environment.yml file:

conda env create -f environment.yml
conda activate splatam

Windows

For Windows installation using Git Bash, refer to the instructions shared in Issue#9.

Docker and Singularity Setup

To run the code inside a docker image, use:

docker pull nkeetha/splatam:v1
bash bash_scripts/start_docker.bash
cd SplaTAM
pip install virtualenv --user
mkdir venv
cd venv
virtualenv --system-site-packages splatam
source ./splatam/bin/activate
pip install -r venv_requirements.txt

Setting up a singularity container follows a similar process:

cd path_to_singularity_folder
singularity pull splatam.sif docker:nkeetha/splatam:v1
singularity instance start --nv splatam.sif splatam
singularity run --nv instance:splatam
cd path_to_SplaTAM
pip install virtualenv --user
mkdir venv
cd venv
virtualenv --system-site-packages splatam
source ./splatam/bin/activate
pip install -r venv_requirements.txt

Online Demo

You can run SplaTAM in your own environment using an iPhone or LiDAR-equipped Apple device. Download the NeRFCapture app, connect your iPhone and PC to the same WiFi network, and run:

bash bash_scripts/online_demo.bash configs/iphone/online_demo.py

Click send for successive frames. After capturing frames, the app will disconnect from your PC. Check out SplaTAM’s interactive rendering of the reconstruction on your PC!

Usage

To use SplaTAM, follow these steps while utilizing the iPhone dataset as an example:

• Run SplaTAM:

  python scripts/splatam.py configs/iphone/splatam.py

• Visualize the final interactive SplaTAM reconstruction:

  python viz_scripts/final_recon.py configs/iphone/splatam.py

• Online visualization:

  python viz_scripts/online_recon.py configs/iphone/splatam.py

• Export splats to a .ply file:

  python scripts/export_ply.py configs/iphone/splatam.py

  Use viewers like SuperSplat or PolyCam for visualization.

• Run 3D Gaussian Splatting:

  python scripts/post_splatam_opt.py configs/iphone/post_splatam_opt.py

• Using ground truth poses:

  python scripts/gaussian_splatting.py configs/iphone/gaussian_splatting.py

Downloads

By default, the DATAROOT is set to .data. Adjust the input folder in your config files as necessary.

• Replica:

  bash bash_scripts/download_replica.sh

• TUM-RGBD:

  bash bash_scripts/download_tum.sh

• ScanNet: Follow the data downloading procedure on the ScanNet website.

Benchmarking

For effective evaluation while running SplaTAM, utilize Weights and Biases. Enable this by setting the wandb flag to True in the config file.

For each scene, adjust the input_folder and output paths in the config files. Check the example commands for various datasets available.

Acknowledgement

We appreciate contributions from open-source repositories that informed our work.

Citation

If you find SplaTAM useful, we encourage you to cite our work:

@inproceedings{keetha2024splatam,
    title={SplaTAM: Splat, Track, Map 3D Gaussians for Dense RGB-D SLAM},
    author={Keetha, Nikhil and Karhade, Jay and Jatavallabhula, Krishna Murthy and Yang, Gengshan and Scherer, Sebastian and Ramanan, Deva and Luiten, Jonathon},
    booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
    year={2024}
}

Developers

Meet the talented minds behind SplaTAM:

• Nik-V9 (Nikhil Keetha)
• JayKarhade (Jay Karhade)
• JonathonLuiten (Jonathon Luiten)
• krrish94 (Krishna Murthy Jatavallabhula)
• gengshan-y (Gengshan Yang)

Troubleshooting

If you run into issues during the installation or usage of SplaTAM, here are a few troubleshooting tips:

• Ensure that you have the correct version of Python and Torch installed that matches your CUDA toolkit.
• If your project requirements fail to install, double-check the requirements.txt file for the necessary packages.
• Running the program inside a virtual environment can help isolate potential clashes with system packages.

For more insights, updates, or to collaborate on AI development projects, stay connected with **fxis.ai**.

At **fxis.ai**, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.

Happy SplaTAMing!