Exploring Gaussian Splatting

A Gaussian splat is the first 3D format that has felt like magic to me since I first rotated a cube in WebGL. You take a video, walk it through some software, and what comes out the other side is a scene you can fly through. Not a mesh, not a photo, not a video. The actual room, reconstructed, that you look around inside of.

Here is one I trained. It started life as a walkthrough video of a farmhouse. Load it and drag to look around. On a laptop, use WASD to move; on a phone, there’s a joystick.

3D · gaussian splat

Farmhouse interior

▶ Load 3D scene

730k Gaussians · 13 MB

What you’re looking at is roughly 730,000 little 3D blobs. Each one is a Gaussian: a position in space, a color, a scale, an orientation, and an opacity. Stack enough of them, sort them back to front, blend them, and you get a continuous-looking surface out of fuzz. The clever part is how they’re made. You don’t model them. You start with a noisy point cloud and run gradient descent, nudging every blob’s position and color and shape until the rendered result matches the photos you fed in. It’s training, the same loop as any other model, except the weights are a scene.

That’s the part that hooked me. I’ve been reconstructing rooms, objects, whole spaces from nothing but a video off my phone, and the results are good enough that I keep showing people and watching them tilt their head.

The GPU problem nobody warns you about

Something almost stopped me before I started. Gaussian splatting came out of academia, and academia runs on NVIDIA. Every reference implementation, gsplat, Nerfstudio, the commercial tools like Postshot, assumes you have a CUDA card. The training kernels are written in CUDA. The whole ecosystem is built on it.

I have an AMD RX 6700 XT. 12 GB of VRAM, great for games, completely locked out of CUDA. On paper I was the wrong person to try this.

What saved me is a project called Brush, a splatting trainer written in Rust on top of wgpu. Because it targets wgpu instead of CUDA directly, it runs on Vulkan, on Metal, on whatever the machine has. That one decision is the difference between “this is a research toy for people with a 4090” and “this runs on the gaming PC under my desk.” I’ve trained scenes on the AMD card and on an M1 Max MacBook, neither of which can run a single line of the standard toolchain.

It is not frictionless. The sharpest example: wgpu will silently fall back to llvmpipe, a software renderer, if you don’t pin the backend. The training “works”, it just runs on the CPU at maybe five percent of the speed, and nothing tells you. The fix is two environment variables.

WGPU_POWER_PREF=high WGPU_BACKEND=vulkan brush ...

I lost a real evening to that before I noticed the GPU sitting at idle while a “GPU” job ran. The lesson that keeps repeating with AMD: things work, but you have to know exactly which lever to pull, and the defaults are tuned for the hardware you don’t own.

From a phone video to a scene

The pipeline is four steps, and once it’s scripted it’s boring in the best way.

1. Capture. Walk the space slowly with the camera. I’ve used a Pixel 9 Pro at 4K30. Slow and continuous matters more than long, for reasons I’ll get to.
2. Extract frames. ffmpeg pulls frames out of the video, two per second, capped at 2048px on the long side.
3. Solve the cameras. COLMAP runs structure-from-motion: it looks at all those frames and works out where the camera was for each one. This is the step that decides whether you get a scene or a mess.
4. Train. Brush takes the frames and the camera poses and trains the Gaussians, 30,000 steps, about 25 minutes on my card.

The best run I’ve done was the most hobbyist thing imaginable: a 1/64 scale model of a 7-Eleven, the kind of diorama people build for fun. I shot 94 seconds of it on the Pixel, 4K30, orbiting slowly. COLMAP registered 377 of 377 frames, a clean 100 percent. The whole thing, video to finished splat, took 35 minutes and scored higher than any of the room-scale captures I’d done. Small, controlled, well-lit, shot with patience. The diorama taught me more about good capture than any of the big scenes did.

On training length, I’ll save you an experiment. The default in a lot of guides is 50,000 steps. I ran the curve and the gap between 30k and 50k is under half a decibel of PSNR and a thousandth of an SSIM point. Imperceptible. I dropped my default to 30k and got 40 percent of my time back. Past a point you’re just reheating the same scene.

The output is a .ply file, and they’re huge: the bigger scenes come out at several hundred megabytes, a few million Gaussians. To put one in a browser you compress it. The viewer above is a .sog, the same farmhouse squeezed from a couple hundred megabytes down to 13. One quirk worth writing down: on the 6700 XT I have to compress with spherical harmonics capped at degree 1, because the full set crashes the WebGPU encoder on this card. There’s that AMD tax again.

The floaters

Now the part that ate most of my time.

Train a scene and the first thing you notice, after the initial “it worked”, is the junk. Faint translucent sheets hanging in mid-air. Spikes coming off reflective surfaces. A haze where a window should be. These are floaters: Gaussians that the training placed badly, usually because the camera solve was uncertain about that part of the scene, so the optimizer parked some blobs in the wrong place and they happened to reduce the loss.

The maddening thing is how few of them there are. In one scene I dug into, the floaters were about one percent of the total Gaussian count. But a single floater can be a Gaussian scaled to ten meters across, so one percent of the blobs can cover most of what you actually see. In that same scene, around 62,000 Gaussians had been flung so far outside the room that the scene’s bounding box had blown out to 765 meters. A farmhouse, in a box the size of a few city blocks, almost all of it empty except for a sprinkle of garbage.

I tried things that didn’t work, which is the useful part.

Masking the floaters away during training. The idea: paint out the parts of each frame I didn’t want (people walking through, mostly) with an alpha channel, and tell Brush to make those regions transparent. I turned the alpha weight up to 1.0 and the trainer collapsed the entire scene to four Gaussians. Four, from 1.5 million. The reason is obvious in hindsight: a person appears at a different 3D location in every frame, so “make this transparent” became a pile of contradictory constraints across views, and the optimizer’s least-bad answer was to delete almost everything. I deleted the masking instead.

A turntable. For the diorama, my first instinct was to put it on a lazy-Susan, spin it, and hold the camera still. Clean, repeatable. It produced an empty room with a smear in the middle. Structure-from-motion assumes one rigid scene and a moving camera. I’d given it a static room and a moving object, which is two rigid bodies, and COLMAP locked onto the feature-rich background and treated the thing I actually cared about as noise. The fix is geometrically identical and completely different in practice: lock the object, orbit the camera. Same relative motion, but now the math holds.

That failure points at the single biggest lever in the whole process, which is capture, not training. I have two takes of the same diorama. Take A was longer, three minutes, but I’d panned quickly between sections and the motion blur broke COLMAP’s frame-to-frame tracking into six disconnected fragments. Take B was shorter, two and a half minutes, but slow and continuous, and it solved as two clean pieces with twenty times more usable Gaussians. Slower and continuous beats longer with gaps, every time. You’re not filming for a person to watch, you’re filming for an algorithm to triangulate.

What did work was attacking floaters from both ends.

After training, I filter the .ply directly with PlayCanvas’s splat-transform. Two operations: crop to a bounding box so the 765-meter explosion collapses back to the ~20 meters that’s actually the room, and cap the per-Gaussian scale so anything stretched beyond about two meters across gets deleted. It runs in roughly 13 seconds, no GPU, and it removed about 66,000 pieces of junk from that scene without touching the real geometry.

During training, the bigger win is telling the optimizer to be less eager. Dropping spherical harmonics from degree 3 to 2 kills a lot of the view-dependent streaking off shiny surfaces. Nudging up the penalties on Gaussian scale and opacity makes it stop spawning huge faint blobs to paper over uncertainty. Together those cut one scene from 1.55 million Gaussians to 890,000, a 43 percent reduction, with no visible quality loss. A smaller, tighter scene that also happens to load faster. The two approaches stack: train clean, then filter what’s left.

What I still can’t fully beat is reflections. A stainless sink or a mirror produces specular spikes that aren’t mid-air and aren’t oversized, so neither the crop nor the scale cap catches them, and dropping the harmonics only softens it. That one’s still open.

Where this goes

Everything above produces a frozen moment. The farmhouse you’re looking at is one instant, reconstructed. Nothing in it moves.

The frontier is making it move: 4D splatting, where the Gaussians have a time dimension and the scene plays back like volumetric video you can walk around inside. The research is real and moving fast. The catch is capture. To reconstruct a moving scene from a single camera you’d have to be in every place at once, so the groups doing this convincingly build rooms lined with dozens of synchronized cameras firing together, capturing every angle at every instant. That works, and it is completely out of reach for someone with a phone.

But so was the static version, a few years ago. The static reconstruction I’m doing on a gaming GPU today was a lab with a camera rig not long before that. The interesting question isn’t whether 4D-from-a-phone arrives, it’s what I’ll do with it the week it does. The thing I keep coming back to is memory. A photo of a place you loved is a flat window. A splat is the room. A temporal splat would be the afternoon, the light moving, the people in it, something you could step back into instead of look at.

I don’t think that’s far off. And when it lands, it’ll land on consumer hardware first for whoever already knows which lever to pull.