Are you interested in segmenting 3D Gaussians with the SAGA framework? If so, you’re in the right place! In this guide, we’ll walk you through the installation, setup process, and usage of SAGA, making sure you have a smooth sailing experience.

Step 1: Installation

Installing SAGA is as easy as pie. The installation process mirrors that of 3D Gaussian Splatting. Here’s how to do it:

• Clone the SAGA repository:

git clone git@github.com:JumpatSegAnyGaussians.git

• Install the required dependencies:

conda env create --file environment.yml

• Activate the environment:

conda activate gaussian_splatting

For using the public ViT-H model for SAM, download the pre-trained model from here and place it under .third_party/segment-anything/sam_ckpt.

Step 2: Prepare Your Data

The datasets you will be using include:

• 360_v2
• nerf_llff_data
• LERF

The data structure of SAGA should look like this:

.data
    360_v2
        garden
            images
            ...
    nerf_llff_data
        fern
            images
            ...
    lerf_data
        ...

Step 3: Pre-train the 3D Gaussians

To get the most out of SAGA, you first need to pre-train the 3D Gaussians, which is akin to preparing a canvas before painting a masterpiece. Here’s how to do it:

python train_scene.py -s path_to_COLMAP_or_NeRF_Synthetic_dataset

Step 4: Extract Masks and Features

Next, let’s get those sam_masks and corresponding mask scales:

python extract_segment_everything_masks.py --image_root path_to_scene_data --sam_checkpoint_path path_to_pre-trained_SAM_model --downsample 1248
python get_scale.py --image_root path_to_scene_data --model_path path_to_pre-trained_3DGS_model

If limited GPU memory is a concern, the downsample parameter is your best friend!

Step 5: Train 3D Gaussian Affinity Features

Now, it’s time to train the features:

python train_contrastive_feature.py -m path_to_pre-trained_3DGS_model --iterations 10000 --num_sampled_rays 1000

Step 6: Run the 3D Segmentation

SAGA provides an interactive GUI (with saga_gui.py) and a Jupyter notebook (with prompt_segmenting.ipynb) for 3D segmentation. To launch the GUI, simply:

python saga_gui.py --model_path path_to_pre-trained_3DGS_model

If you want to explore open-vocabulary segmentation, check out the Jupyter notebook for detailed instructions.

Understanding the Code: A Creative Analogy

Think of the installation and coding process like preparing a dish in a kitchen. Here’s a fun analogy:

• Ingredients (Data): Just like a cook needs fresh ingredients, you will gather datasets for SAGA to process and perform segmentation.
• Recipe (Code): Each code command is a step in the recipe. You carefully prepare and follow them to create your dish, or in this case, segment 3D models.
• Cooking (Execution): Finally, using the GUI is like tasting your dish to check if it’s seasoned properly. You can adjust the parameters and add more to enhance the flavor (or your segmentation results) as needed.

Troubleshooting

If you encounter any issues, consider the following tips:

• Ensure that all paths are accurately set based on your folder structure.
• Check your GPU memory; higher models may require more memory.
• If the GUI is unresponsive, consider running less intensive operations or reducing the input size.
• For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.

Rendering Results

After you’ve saved your segmentation results, run the following command to render them:

python render.py -m path_to_pre-trained_3DGS_model --precomputed_mask path_to_segmentation_results --target scene

Conclusion

Congratulations! You’ve successfully installed SAGA and are now on your way to segmenting 3D Gaussians like a pro. Keep experimenting, and enjoy the fascinating world of 3D segmentation!