Real-Time 3D as a UX Medium: Beyond Visualization

Working with Zoox's 3D perception stack changed my understanding of what the medium can do when it's directly coupled to a live sensor pipeline.

The difference between visualisation and representation

A traditional visualisation takes data and represents it visually: a chart, a map, a diagram. A real-time 3D perception display does something different: it reconstructs a world. When you're looking at a LiDAR point cloud of a city block, you're not looking at a representation of sensor data. You're looking at the sensor's understanding of space. The design problem is not how to show the data, it's how to show what the machine perceives, so the human can evaluate whether that perception is correct.

Colour as semantics

In our 3D tooling, colour in the point cloud wasn't decorative; it carried semantic meaning. This object is classified as a pedestrian. This region is uncertain. This detection disagrees with the map. Designing this semantic colour system required close collaboration with the perception engineering team, because every colour decision was also a claim about how the system understood the world. Getting it wrong didn't just look bad; it created false confidence or unnecessary alarm in operators using the tool.

Time is a spatial dimension

In real-time 3D, the fourth dimension, time is often the most important. Showing where an object is right now matters less than showing where it's going. Showing what the sensor sees now matters less than showing what it saw a moment ago and how that's changed. Designing for temporal continuity in a 3D environment requires thinking about decay, prediction, and history as fundamental visual primitives, not enhancements.

The 3D environment as a validation tool

At Zoox, the 3D perception tools weren't just for operators; they were for engineers validating autonomy software, researchers for collecting data and behavior, or riders to learn more about the autonomy. They needed to be able to load a scenario, scrub through time, overlay ground truth against predictions, and identify exactly where the perception stack was failing. This is a very different UX problem from a consumer-facing 3D experience: the user is an expert, they're debugging a system, and the interface needs to support a forensic, investigative mode of interaction that most 3D environments don't consider.

Real-time 3D is becoming a design discipline

As autonomous systems, spatial computing, and AI-driven environments proliferate, real-time 3D is becoming a mainstream design surface. The skills developed in designing for perception stacks, sensor fusion, and live spatial data are directly applicable to AR/VR interfaces, digital twin environments, and the next generation of AI tools that need to communicate spatial understanding. If you're a UX designer who hasn't yet engaged seriously with 3D as a medium, now is the time to start.