Sampling and Selection Methods for Applying 2D Neural Networks to 3D Gaussian Splats

Abstract

We propose a novel approach for applying interpolation methods to unstructured volumetric data that allows for the operation of 2D neural networks directly on 3D Gaussian splats. Gaussian splatting is at the cutting edge of volume rendering methods, 2D neural networks have achieved a dominant and lasting degree of success and real-life application. We propose leveraging the advantages of both, an approach which is the first of its kind. We extend the method for interpolated convolution on 3D surface meshes with 2D CNNs by Hart et al to the unstructured 3D volumetric data of Gaussian splats and present an end-to-end pipeline for our work. We showcase our results with style transfers on 3D Gaussian splats performed by a 2D convolution model with no retraining. Our results compare favorably with those of current approaches to performing style transfers on 3D Gaussians using purpose-built and purpose-trained 3D models.