A sorting-free formulation for ray-traced 3DGS optimization, enabling efficient reconstruction and rendering of both standard and relightable 3DGS scenes.
1University of Illinois at Urbana-Champaign | 2Adobe Research
Ray-tracing-based 3D Gaussian splatting (3DGS) enjoys the generality of supporting non-pinhole camera models and relightable formulations. However, they are usually lacking in performance, partially due to the need for depth-based sorting of all intersecting Gaussians along the traced rays.
In this paper, we introduce a sorting-free differentiable stochastic formulation for ray-traced 3DGS, enabling efficient reconstruction and rendering of both standard and relightable 3DGS scenes. For standard 3DGS, our method offers performance comparable to rasterization-based 3DGS and outperforms sorting-based ray tracing. For relightable 3DGS, our technique provides higher-quality reconstructions and renderings thanks to the accurate shadow and shading computation provided by per-Gaussian shading via fully ray-traced shadow rays.
As a ray-tracing-based algorithm, our method brings a considerable speed-up of ~40-50% to 3DGRT, and runs in similar speed to the rasterization-based 3DGS. Meanwhile, our algorithm achieves comparable reconstruction quality on all benchmarks.
Drag the timeline or press play to compare optimization progress across methods at equal wall-clock time. Drag the vertical handles to adjust the comparison split.
Furthermore, our method can be applied to relightable Gaussian Splatting with a simple NEE-style extension. Our algorithm can be used to efficiently support ray-traced shadows and various types of emitters (non-distant/area/image-based/...), and achieve state-of-the-art results.
Animated relighting under rotating environment lighting. Each cell shows the ground-truth reference (left) alongside the relighted prediction (right). Drag the slider to control the light direction.
The core of our method is a two-sample Monte Carlo estimator that unbiasedly estimates the radiance and the gradient for each gaussian. Our method provides two important benefits: 1. The Gaussians along the traced rays can be processed in arbitrary order, removing the need for sorting all Gaussians along each traced rays, and 2. only a small subset of gaussians will contribute to the pixel color or receive gradient, reducing the cost of Gaussian color evaluation. We further show that our algorithm provides proper importance sampling that is sufficient to avoid loss of quality.
If you find this work useful, please cite our paper: