tested on Python 3.9.19, CUDA 11.8

1. Clone the repository

git clone --recurse-submodules https://github.com/branyang02/gsplat.git
cd gsplat

2. Create a new conda environment

conda env create -f environment.yml && conda activaate semantic_gen

3. Install gsplat and segment-anything-langsplat

pip install -e .
pip install -e segment-anything-langsplat

4. Navigate to semantic_gen directory and install dependencies

cd 3d_semantic && pip install -r requirements.txt

5. Download SAM checkpoint

mkdir -p ckpts && wget -P ckpts https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth

All scripts should be run from the semantic_gen directory.

Download Mip-NeRF 360 dataset:

python datasets/download_dataset.py

First, we extract CLIP embeddings for the training and validation sets.

python preprocess.py --data_dir data/[dataset_dir] --sam_ckpt [sam_ckpt] --num_workers 8

Adjust num_workers based on your GPU's capacity. Data are processed in parallel to speed up the process.

This saves the precomputed embeddings in data/[dataset_dir]/trainset and data/[dataset_dir]/valset.

python sam_trainer.py --data_dir data/[dataset_dir] --result_dir results/[dataset_dir]

python sam_viewer.py --ckpt results/[dataset_dir]/ckpts/[ckpt]

http://www.gsplat.studio/

gsplat is an open-source library for CUDA accelerated rasterization of gaussians with python bindings. It is inspired by the SIGGRAPH paper 3D Gaussian Splatting for Real-Time Rendering of Radiance Fields, but we’ve made gsplat even faster, more memory efficient, and with a growing list of new features!

Dependence: Please install Pytorch first.

The easiest way is to install from PyPI. In this way it will build the CUDA code on the first run (JIT).

pip install gsplat

Or install from source. In this way it will build the CUDA code during installation.

pip install git+https://github.com/nerfstudio-project/gsplat.git

To install gsplat on Windows, please check this instruction.

This repo comes with a standalone script that reproduces the official Gaussian Splatting with exactly the same performance on PSNR, SSIM, LPIPS, and converged number of Gaussians. Powered by gsplat’s efficient CUDA implementation, the training takes up to 4x less GPU memory with up to 15% less time to finish than the official implementation. Full report can be found here.

# under examples/
pip install -r requirements.txt
# download mipnerf_360 benchmark data
python datasets/download_dataset.py
# run batch evaluation
bash benchmark.sh

We provide a set of examples to get you started! Below you can find the details about the examples (requires to install some exta dependences via pip install -r examples/requirements.txt)

• Train a 3D Gaussian splatting model on a COLMAP capture.
• Fit a 2D image with 3D Gaussians.
• Render a large scene in real-time.

This repository was born from the curiosity of people on the Nerfstudio team trying to understand a new rendering technique. We welcome contributions of any kind and are open to feedback, bug-reports, and improvements to help expand the capabilities of this software.

This project is developed by the following wonderful contributors (unordered):

• Angjoo Kanazawa (UC Berkeley): Mentor of the project.
• Matthew Tancik (Luma AI): Mentor of the project.
• Vickie Ye (UC Berkeley): Project lead. v0.1 lead.
• Matias Turkulainen (Aalto University): Core developer.
• Ruilong Li (UC Berkeley): Core developer. v1.0 lead.
• Justin Kerr (UC Berkeley): Core developer.
• Brent Yi (UC Berkeley): Core developer.
• Zhuoyang Pan (ShanghaiTech University): Core developer.
• Jianbo Ye (Amazon): Core developer.

We also have made the mathematical supplement, with conventions and derivations, available here. If you find this library useful in your projects or papers, please consider citing:

@misc{ye2023mathematical,
    title={Mathematical Supplement for the $\texttt{gsplat}$ Library},
    author={Vickie Ye and Angjoo Kanazawa},
    year={2023},
    eprint={2312.02121},
    archivePrefix={arXiv},
    primaryClass={cs.MS}
}

We welcome contributions of any kind and are open to feedback, bug-reports, and improvements to help expand the capabilities of this software. Please check docs/DEV.md for more info about development.