Welcome to the exciting world of real-time radiance field rendering! In this blog, we will explore an unofficial implementation of the 3D Gaussian Splatting algorithm using Taichi, a high-performance computing language. This blog will guide you through the installation process, and training the model, as well as troubleshooting common issues you may encounter along the way.

What is 3D Gaussian Splatting?

The 3D Gaussian Splatting algorithm plays a pivotal role in rendering point clouds gathered from multiple viewpoints. It transforms images from these viewpoints into a denser representation of a scene, including rich details and attributes like covariance and color information. This process occurs in two steps: Training and Inference.

Understanding the Training Process

During the training phase, the algorithm ingests images from various angles, a sparse point cloud, and camera pose data. It utilizes a differentiable rasterizer to refine the point cloud, ultimately creating a dense point cloud packed with extra features. This process can be compared to an artist refining a rough sketch into a detailed masterpiece, where each iteration adds depth and color to the initial outline.

graph LR
    A[ImageFromMultiViews] -- B((Training))
    C[sparsePointCloud] -- B
    D[CameraPose] -- B
    B -- E[DensePointCloudWithExtraFeatures]

Understanding the Inference Process

For the inference phase, the algorithm leverages the dense point cloud produced in the training phase, along with any arbitrary camera pose, to render an image. Imagine picking a slice of a cake (point cloud) and using it to create a delicious visual dessert (rendered image) that looks appealing from any angle.

graph LR
    C[DensePointCloudWithExtraFeatures] -- B((Inference))
    D[NewCameraPose] -- B
    B -- E[Image]

Installation Steps

To get started with 3D Gaussian Splatting, follow the steps below:

1. Set up an environment that contains PyTorch and torchvision.
2. Clone the repository and navigate to its directory.
3. Run the following commands:

pip install -r requirements.txt
pip install -e .

Make sure you are running the code on an environment like Ubuntu 20.04.2 LTS with Python 3.10.10 and equipped with an RTX 3090 and CUDA 12.1.

Working with Datasets

The algorithm requires a point cloud for the entire scene, camera parameters, and ground truth images. You can build the dataset using various formats, such as JSON for camera parameters and Parquet for point clouds. Below is a typical folder structure you should follow:

• data
• tat_truck_every_8_test
• train.json
• val.json
• point_cloud.parquet
• image
• 000000.png
• 000001.png

Training the Model

Once your dataset is ready, you can begin training by executing:

bash python gaussian_point_train.py --train_config configtat_truck_every_8_test.yaml

Troubleshooting Tips

While working with the 3D Gaussian Splatting implementation, you might encounter some issues. Here are a few troubleshooting tips:

• If you experience slow training speeds, ensure that your dataset is properly configured and make use of smaller images for faster processing.
• Check your CUDA compatibility and update drivers if necessary.
• For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

Conclusion

At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.