Bypassing the Mesh: Gaussian Splatting in the 2026 Pipeline

If the return to projectors and green screens represents Hollywood's desire for cheaper physical infrastructure, 3D Gaussian Splatting is the mathematical engine making that transition artistically viable.

How We Got Here

Photogrammetry has been the backbone of 3D capture for decades. Take enough overlapping photographs of a subject, find the matching points across images, calculate the geometry, project the textures onto a mesh. It works reliably on hard surfaces and is well understood across architecture, archaeology, VFX, and game production.

The limitations are equally well known. Complex surfaces — glass, water, foliage, anything reflective or translucent — don't produce reliable matching points. The raw mesh output is dense and irregular, requiring significant retopology work before it is usable in a production pipeline. The result is also baked to the lighting conditions at the time of capture — change the light and the asset no longer matches.

Neural Radiance Fields (NeRFs)

In 2020, a UC Berkeley research paper introduced Neural Radiance Fields (NeRFs). Rather than constructing a polygon mesh, a neural network learns the geometry and light behaviour of a space well enough to render it from angles that were never directly photographed.

The practical improvement over photogrammetry was significant on complex materials. View-dependent effects — reflections that shift as you move, translucent surfaces, fine lighting gradients — are handled by the network rather than approximated through texture projection. No mesh cleanup, no retopology.

The gains: more photorealistic results on complex surfaces and materials; reflections, transparency, and fine detail that photogrammetry typically distorts or loses; novel view synthesis — rendering angles the original camera never captured.

The limitations: rendering a single frame required querying the neural network thousands of times per pixel — minutes per frame at training time, with significant inference overhead throughout. Real-time use was not achievable on available hardware.

The more fundamental constraint was editability. A neural network stores the scene implicitly — you cannot inspect or modify what it has learned directly. Removing an object, changing the lighting, or adjusting any element meant retraining or working around the network. For production environments requiring fast iteration, this limited practical usefulness.

Between 2021 and 2022, variants addressed specific bottlenecks. Instant-NGP reduced training times considerably through hash-based encoding. Block-NeRF extended the approach to large outdoor environments. These were meaningful improvements within the same fundamental architecture.

3D Gaussian Splatting

The 2023 paper from INRIA — Kerbl et al. — approached the problem differently. Rather than a neural network, 3DGS stores the scene as an explicit collection of millions of small ellipsoids — Gaussians — each defined by its position, covariance, colour, and opacity.

Rendering is done by rasterising these Gaussians onto the image plane, a process that maps well to GPU parallelism and avoids neural inference entirely. Render speeds above 100 frames per second on standard hardware became achievable — the threshold required for real-time LED volume work, where the background must track precisely with a moving physical camera.

The scene is an explicit data structure rather than an encoded network, so individual elements can be identified, modified, or removed without retraining. Relighting pipelines — separating appearance from lighting data — became architecturally possible in a way they had not been with NeRF.

The shift from NeRF to 3DGS is not primarily a speed improvement. It is a change in what the underlying data structure allows — and that is what opened the door to production use.

From Months of Modelling to Hours of Scanning

Until recently, virtual production environments were entirely beholden to traditional 3D pipelines: importing heavy polygonal meshes, manually baking textures, and wrestling with frame-rate optimisation inside Unreal Engine. It was a slow, artist-intensive bottleneck.

In 2026, the pipeline has broken clean of the polygon. 3D Gaussian Splatting (3DGS) has transitioned from an exotic research curiosity into a standardised enterprise pipeline asset.

3DGS completely upends how digital environments are captured and stored. Instead of laboriously constructing a digital environment asset by asset, a VFX crew can visit a physical location, record a few minutes of high-resolution video via a drone or smartphone, and allow a cloud-based algorithm to reconstruct the scene as millions of tiny, transparent, light-emitting points.

The structural impact on film budgets is significant. The traditional Virtual Art Department pipeline shrinks from months to hours. Real-world architectural spaces, rugged terrain, and complex lighting conditions are captured with photorealism, bypassing the need for extensive 3D asset libraries.

The market numbers reflect the pace of adoption. The global virtual production market sat at approximately $2.9 billion in 2025; estimates project $18.5 billion by 2035. The budget impact at the production level is immediate — utilising a 3DGS-scanned environment on an LED volume stage can save a production anywhere from $500,000 to over $5 million compared to building physical sets or moving a full crew to a remote location.

Gaussian Splats are rendered as points rather than complex geometric meshes requiring real-time ray-tracing, which means they run at over 100 frames per second. This speed makes them ideal for background projection layers and live camera-tracking setups on standard, lower-cost workstation hardware.

The Shift from Chaos to Control

The wider AI conversation around filmmaking has been dominated by generative video tools — systems that produce striking output from a text prompt but cannot hold spatial consistency across camera angles, replicate a shot from a different lens position, or integrate cleanly into a professional compositing pipeline. The results are often impressive in isolation and impractical the moment any predictable, repeatable workflow is required. Prompt engineering is a probabilistic exercise. The output varies. The process cannot be engineered.

3D Gaussian Splatting inverts that proposition entirely. It is a deterministic spatial engineering tool. The data it captures is explicit: exact real-world dimensions, spatial relationships, and view-dependent light behaviour recorded from physical space and stored as an inspectable data structure. Nothing is guessed. Nothing is inferred by a closed-box network. Artists can examine, clean, crop, and manipulate the data directly, passing it into standard production tools — Houdini for volumetric manipulation, Unreal Engine for real-time deployment.

The real industry transformation is not AI making arbitrary creative decisions on behalf of directors. It is AI building a photorealistic, spatially accurate substrate that preserves human creative control over the lens, camera movement, and lighting. Moving from field scan to LED wall — bypassing mesh reconstruction entirely — is not an experimental workflow. It is a line item on a production budget.

For a practical demonstration of how single-image explicit spatial point workflows operate without requiring mesh topology or heavy local compute, How to Create an Interactive 3D Gaussian Splat From 1 Image (2026) walks through the full setup.

Industrial Validation: The 2026 Toolkit

The shift from experimental workflow to core infrastructure is marked by a wave of major industry adoptions.

Platform / Tool	Integration	Pipeline Function
Capture
XGRIDS L2 Pro	Hardware LiDAR/SLAM Scanner	Combines LiDAR with multi-SLAM algorithms, building a millimetre-detailed point cloud in real time as the operator walks the space. Designed for high-precision location capture feeding directly into GS processing pipelines.
DJI Terra	Enterprise Drone Processing	Transforms aerial drone imagery directly into Gaussian Splatting PLY files — the standard enterprise entry point for outdoor and large-scale environment capture.
Luma AI	UE5 Plugin + Mobile Capture	End-to-end platform from mobile capture to Unreal Engine integration. Widely adopted across creative pipelines from consumer to professional; UE5 plugin deploys processed splats directly into the production environment.
Process & Build
Volinga	Cloud Processing + Relightable Splats	Cloud-based processing pipeline and UE5 plugin. Handles splat optimisation, count reduction for real-time performance, and relightable splat decomposition — separating the appearance layer from lighting for on-set DP adjustment without recapture.
SuperSplat	Web-Based Cleanup	Browser-based tool for cropping splat boundaries and removing floaters before environments enter the stage pipeline. No local installation required.
SplatForge	Blender Add-on	Bridges 3DGS into classic VFX pipelines — colour grading and scene manipulation of 16M+ splat environments inside Blender.
Compositing & VFX
OTOY OctaneRender	Full Path-Traced 3DGS	Hardware-accelerated, path-traced splat rendering — bridging point-cloud capture and cinematic global illumination.
Arvid Schneider / CG Lounge	Houdini GSOPs + OpenUSD	VFX Supervisor documentation of integrating 3DGS with Houdini's GSOP nodes and OpenUSD — specifically tackling shadows and global illumination between traditional polygon assets and splat backgrounds.
World Labs (Marble)	Generative Spatial AI	Generates complete, geometrically sound 3D environments from text prompts or 2D concept sketches, exported directly as interactive splats.
Cinema 4D	Gaussian Splat Studio Plugin	Imports, views, renders, and exports splat data directly inside C4D with Redshift and Standard/Physical material support. Non-destructive procedural workflow — MoGraph effectors and fields enable animation and distortion of splat geometry without rebuilding scenes.
Adobe After Effects	Gaussian Splatting (AE Scripts)	Imports .PLY splat files directly into After Effects — a format AE cannot handle natively. Renders at speed inside the standard AE compositing pipeline, bringing volumetric captures into motion graphics, broadcast, and title sequence work without round-tripping through a 3D app.
Live Production & Stage
Sony XYN	Spatial Capture Solution	AR-guided mirrorless camera scanning via a companion iOS app, cloud processing into high-fidelity 3DCG assets, and a UE5 renderer plugin optimised for LED volume deployment. Built to handle reflective surfaces, transparent materials, and complex lighting conditions that traditional photogrammetry cannot reliably reproduce.
Disguise / RenderStream	Media Server + LED Volume Hub	Brings Sony XYN, Unreal Engine, Volinga, Omniverse, TouchDesigner, Notch, and Unity into a unified show environment. RenderStream manages the real-time closed loop between physical camera tracking, rendering engine, and LED display. OpenUSD ingestion handled via the RenderStream-RTX pipeline using an NVIDIA Omniverse Kit subsystem.
Lightcraft Spark	Indie Virtual Production	Connects script databases with live iOS virtual tracking, USD downstream workflows, and Gaussian Splat rendering. Runs without a dedicated media server — full virtual production on standard hardware at indie budget scale.
Streaming & Web
Cesium for Unreal	3D Tiles Streaming Architecture	Streams Gaussian Splats into Unreal Engine via the Cesium ion pipeline using KHR_gaussian_splatting and SPZ compression. HLOD delivers sub-centimetre visual fidelity at the camera position while down-sampling background clusters — eliminating VRAM overload from monolithic .ply loading.
Aholo Viewer	Open-Source Browser Renderer	High-performance renderer designed for 3DGS and meshes. Chunked streaming Level of Detail handles massive splat environments directly in a standard web browser — no local workstation hardware required.
Consumer & XR
SHARP (Apple)	Single-Image Splat Generation	Open-source model generating a Gaussian Splat from a single photograph in under one second. Deployed at scale in Apple Maps (WWDC 2026) and foundational to Vision Pro's spatial memory capabilities. Reconstructs depth and geometry within the captured field of view.
Feature Deployments
Superman (2025)	4D Gaussian Splatting	Framestore deployed 4DGS for the holographic sequences of Jor-El and Lara Lor-Van in the Fortress of Solitude (~40 shots). Infinite Realities' 192-camera volumetric rig handled actor capture; GSOPs in Houdini handled downstream manipulation and pruning of high-density temporal splat arrays.

The budget spectrum of adoption is telling. At the top end, Framestore structured the 4DGS pipeline on James Gunn's Superman around a clear division of labour for the holographic sequences of Jor-El and Lara Lor-Van in the Fortress of Solitude (~40 shots). Infinite Realities' 192-camera volumetric capture rig collected actor data; downstream, SideFX Houdini with open-source GSOPs manipulated, animated, and pruned high-density temporal splat arrays to give the director complete post-production camera freedom.

At the mid-tier, Studio Lab — under Virtual Production Director Ian Messina, working alongside Dimension Five and CG Pro — is establishing the operational playbook for how commercial and independent productions skip mesh construction entirely and move directly from scan to LED stage. THE FUTURE 3D specialises in exactly this Scan-to-LED workflow, combining DJI Matrice aerial passes with XGRIDS L2 Pro handheld scanning to build production-ready LED wall environments. Digital Film Tree takes a different approach — advocating for 3DGS as capture insurance: scan every restricted or expensive location immediately so that pickup shots, VFX extensions, and downstream compositing can be handled without a costly return. The pipeline is not a Hollywood luxury. It is being used to solve real-world, fast-turnaround production problems across budget levels right now.

At the furthest end of that spectrum, workflow case studies from Lightcraft Technology founder Eliot Mack demonstrate how indie filmmakers are pairing mobile camera tracking, real-time AI relighting, and 3DGS-captured backdrops on budgets that would not cover a single day of traditional virtual production. Lightcraft Spark connects script databases directly with live iOS virtual tracking, USD downstream workflows, and Gaussian Splat rendering — running the full virtual production loop on standard hardware without a dedicated media server. For a full breakdown of the platform and its pipeline: Lightcraft Spark: The Browser-Based Tool Reinventing Film Previs. For the NAB 2026 feature overview: This Could Change Previs Forever | Lightcraft Spark at NAB 2026.

The Holy Grail: Relightable Splats

The final barrier to total 3DGS dominance was solved in early 2026 with the commercial deployment of Relightable Gaussian Splatting.

Historically, a splat captured the lighting of the environment exactly as it was shot — scan an alleyway at noon and the shadows were baked in permanently. New virtual production frameworks now successfully decompose the splat into a fixed appearance layer and a variable lighting component. By parameterising the splat primitives to read ambient light metadata, the digital environment can instantly react to changes made by the Director of Photography on set.

When paired with a smart green screen or a laser projector, a filmmaker can capture an environment on a phone, change a noon scene to a stormy night on a laptop, and let a neural network handle the actor re-lighting in post-production. The pipeline is no longer a sequential chain of disjointed departments. It is a unified, real-time feedback loop.

The research foundation for this approach was formalised in GS-ID (ICCV 2025), developed by researchers at HKUST. GS-ID introduced diffusion-guided material priors and spatially-varying spherical Gaussian mixtures to fully disentangle geometry, material, and lighting from a captured asset — allowing complete relighting and editing by modifying light parameters alone, with no retraining required.

The Production Pipeline

Understanding where 3DGS fits requires seeing it as a pipeline with four distinct stages, each now served by purpose-built tooling.

1. Location Scout

Before any capture takes place, the location is assessed — not just physically but as a data collection problem. What surfaces will produce reliable splats? Where are the lighting conditions likely to bake in unwanted shadows? For complex or remote locations, existing aerial imagery and preliminary drone passes inform the capture plan. Increasingly, production teams review splats of similar environments before travelling — using shared volumetric environments to make preliminary creative decisions spatially rather than from photographs.

The logistical impact scales quickly. A director and heads of department can block out scenes, plan complex camera moves, and take accurate measurements of a physical set from thousands of miles away before any crew member travels. The elimination of repeated scouting trips reduces both pre-production budget and carbon footprint at a meaningful scale.

The same spatial data transforms what previsualisation means. Traditional previs has always involved a trade-off between speed and fidelity — grey-box geometry that can be produced quickly but tells a director little about how the real location will photograph. 3DGS-based previs eliminates that trade-off. View-dependent lighting, translucency, and complex material behaviour are present from the first preview pass, rendered interactively on standard GPU hardware and passed downstream into the final VFX pipeline without a separate asset conversion step.

2. Capture

The capture stage has changed significantly. A location that previously required a full survey crew and weeks of photogrammetry scanning can now be captured by a single operator in hours.

A modern capture is a multi-sensor effort. Aerial coverage — building facades, rooflines, wide environmental context — is handled by drone platforms such as the DJI Matrice flying a Zenmuse P1 45MP full-frame sensor, capable of the resolution required for LED volume work. Ground-level spaces, interiors, and complex structural geometry are captured separately using handheld LiDAR/SLAM scanners such as the XGRIDS L2 Pro, which builds a millimetre-detailed point cloud in real time as the operator walks the space. For smaller-scale or consumer applications, smartphones running Luma AI or Polycam produce usable results in minutes.

For high-end productions where set construction or precise match-moving is involved, a hybrid approach adds a terrestrial laser scanner — such as a Trimble X12 — alongside the GS capture. Benchmark testing by THE FUTURE 3D has quantified the precision of standalone GS configurations against high-end terrestrial LiDAR scans: a mean geometric variance of 7.8cm (σ 11.49cm). Sufficient for background environments and rapid set extensions, but short of the sub-millimetre structural accuracy required for physical construction reference. The Trimble scan locks that spatial framework independently, running in tandem with the GS workflow.

3. Process and Build

Raw capture data is processed into a usable Gaussian Splat through platforms such as Volinga and Postshot. This stage includes optimisation — reducing splat count for real-time performance — and, critically, relighting. Relightable pipelines decompose the splat into its appearance layer and its lighting component separately, allowing the DP to treat the captured environment as a variable rather than a fixed condition. A location shot at noon can be adjusted to golden hour or overcast without recapture.

Before a splat enters the stage pipeline, a cleanup pass removes boundary artefacts and floating geometry — floaters produced by reflective surfaces or incomplete coverage during capture. SuperSplat (web-based) handles boundary cropping and artefact removal without requiring local installation. For productions integrating 3DGS into existing VFX pipelines, SplatForge — a Blender add-on — allows artists to colour-grade and manipulate scenes containing over 16 million splats inside Blender, bridging the splat environment with traditional polygon-based asset work.

Mononoke Forest, Lake Shiraoma, Japan — 1,373 captures, viewed in SuperSplat

4. Deployment — Live Events, Stage, and Filmmaking

The processed splat is deployed into the production environment via Unreal Engine 5, using integrations from Volinga, XGRIDS, or Luma AI. On an LED volume stage, the splat runs at 100+ frames per second and responds to the camera tracking system in real time — the background shifts in correct perspective as the physical camera moves.

Disguise functions as the orchestration layer for live production. RenderStream, its proprietary streaming protocol, bridges engines including Unreal Engine, Unity, TouchDesigner, and Notch alongside capture pipelines like Volinga and Sony XYN into a single show environment. Where OpenUSD data is involved, ingestion is currently managed via the RenderStream-RTX pipeline using an NVIDIA Omniverse Kit subsystem, a beta integration that parses and renders USD scenes live on the stage network synchronously, outside of Disguise's core timeline architecture.

The practical result is a genuinely bidirectional data loop. Physical camera telemetry from tracking systems such as Mo-Sys feeds through Disguise into the rendering engine in real time, with focal length, position, and orientation locking the virtual background to the physical lens. RenderStream exposes these parameters back to the operator interface, keeping the handshake between camera and content continuous.

From the operator's position, lighting values, scene variables, and atmospheric parameters are all adjustable from the desk in sync with the performance, without touching the rendering engine directly. The stage show is driven by the operator.

Where It Doesn't Apply

An accurate picture of 3DGS in production includes its boundaries.

Gaussian Splatting is a background layer tool. It excels at environments, architecture, terrain, and atmospheric conditions — everything behind the actors. It does not replace the practical foreground elements that performers physically interact with. Props, set dressing, and any surface an actor touches remain physical or traditionally modelled assets. The splat is the world; the foreground is still built.

Splats are also static representations. They do not natively support real-time physics simulation or dynamic structural interaction. An explosion, a collapsing wall, or any destructive event that needs to interact with the environment requires traditional VFX methods. 3DGS captures reality as it was — it does not simulate how reality changes.

The Expectation Gap

Managing the transition to 3DGS on set involves a psychological shift that is frequently underestimated. The .ply format has a history in structured geometric point clouds — the kind used in autonomous vehicle mapping to identify physical boundaries. Artists and directors encountering a .ply file for the first time bring those expectations with them: crisp surface edges, predictable texture layouts, inspectable geometry.

A Gaussian Splat offers none of that. The asset is not a surface. It is an explicit collection of direction-dependent light behaviours, stored as Spherical Harmonics. What it produces is a volumetric illusion that is entirely dependent on the rendering context in which it is viewed.

The viewport split is where this becomes a production problem. A splat environment running through Unreal Engine's nDisplay on an LED wall at 100+ frames per second looks indistinguishable from a photographed location — the engine is calculating transparency and camera-perspective mathematics correctly in real time. Open that same .ply file in a traditional VFX viewport and the environment appears as a grainy, discontinuous cloud of coloured dots riddled with floating artefacts. Both views are technically accurate representations of the same asset. Neither prepares a first-time user for the other.

The close-up boundary is equally important to communicate before a shoot. In a conventional pipeline, a texture is a 2D image wrapped around solid geometry — zooming in reduces resolution but the surface holds. A Gaussian Splat has no underlying mesh. Move the physical camera past the working distance the environment was designed for and the illusion dissolves into large, semi-transparent blobs. There is nothing beneath the light field to resolve.

Understanding the asset as illusion rather than geometry is the steepest learning curve currently facing virtual art departments adopting this pipeline — and the one most likely to produce on-set frustration if it is not addressed in pre-production.

The Algorithm Is Still Evolving

The 2023 INRIA paper was the starting point, not the fixed standard. Since publication, 3DGS has branched into a family of specialisations, each addressing a different limitation of the base algorithm.

Mip-Splatting improved anti-aliasing and scale consistency — the original paper produced artefacts when viewing splats at varying distances. Scaffold-GS introduced a more efficient scene structure for large environments. 2DGS flattened the ellipsoids into discs, producing cleaner surface geometry for applications that require a mesh output downstream. LangSplat embedded language semantics directly into the splat structure — enabling editors to select and modify scene elements by description rather than manual selection.

Vendor implementations layer proprietary optimisations on top of the base algorithm. Sony XYN, Volinga, and XGRIDS are all working from the same foundational mathematics but solving for different production contexts — volumetric studio capture, real-time relighting, and high-precision field scanning respectively.

The Frontier: Intelligence, Tracking, and Material Understanding

The 2025–2026 research cycle has shifted focus from rendering speed to a more ambitious set of questions: can the splat environment track its own camera in real time, understand the materials it contains, and be queried in natural language? Three papers define the current frontier.

SDGS (CVPR 2026) — Spatial Difference Guided Gaussian Splatting for Simultaneous Localization and 3D Reconstruction — addresses the brittleness of traditional 3DGS against rapid camera movement. Standard 3DGS requires pre-computed camera calibration via tools such as COLMAP before reconstruction begins. Extreme motion blur, high dynamic range transitions, or unstable handheld movement breaks the system. SDGS builds an online, edge-guided framework instead, pairing a standard RGB stream with a hybrid pixel vision sensor that tracks real-time spatial-difference edge signals. The algorithm estimates camera position while simultaneously building a sparse, edge-aligned Gaussian map. Calibration latency is eliminated. A drone or handheld rig encountering dramatic lighting changes can maintain tracking and reconstruction continuously without dropping frame rate.

SplatTalk (ICCV 2025) — 3D VQA with Gaussian Splatting — connects the splat environment directly to language models. The conventional approach to feeding 3D scenes into a Vision-Language Model required first rendering the environment into multi-angle 2D photographs, then letting the model infer spatial relationships from flat images. SplatTalk converts the explicit geometric and colour metadata of the 3DGS primitives directly into 3D visual tokens, feeding them into a pretrained Large Language Model without the rendering intermediary. The practical implication: a production team can query a location scan natively — "Where is the optimal placement for the physical key light to match the virtual sunset?" or "Is there enough clearance for a 15-foot crane in this corridor?" — without manually reviewing rendered previews.

GS-2M (Eurographics 2026) — Material-aware Gaussian Splatting for High-fidelity Mesh Reconstruction — attacks the Expectation Gap described earlier. Standard Gaussian optimisation treats all surfaces equally, producing blobs that collapse into grainy geometry when extracted as a mesh. GS-2M embeds a material-aware layer directly into the optimisation pipeline — identifying material transitions such as glass against matte stone or fabric against metal — and regulates the shape of the Gaussian primitives accordingly. When the asset is exported downstream to a traditional VFX pipeline, localised mesh extraction produces clean, physically coherent surface topology rather than the unorganised geometry that splat-to-mesh conversion typically yields.

Taken together, these three papers describe a different kind of asset: one that tracks its own camera position in real time, understands the materials it contains, and can be interrogated in natural language. The research has moved beyond treating 3DGS as a photogrammetry replacement. It is being built into a complete spatial computing substrate.

The File Format Problem

While the pipeline is maturing rapidly, the industry is still largely shipping environments as raw PLY files — a decades-old polygon format that was never designed for Gaussian Splatting. The field hijacked it out of pragmatism: instead of vertices for a mesh, each point carries XYZ coordinates, rotation, scaling, and Spherical Harmonics data encoding view-dependent colour. It works. It is also completely uncompressed, carrying no scene metadata — no camera rigs, no lighting hierarchy, no animation data. A single high-fidelity location scan can exceed several gigabytes. Streaming it is impractical. Managing it on set is ungainly.

The Khronos Group's KHR_gaussian_splatting extension for glTF 2.0 — developed in collaboration with Niantic Spatial, Cesium, and Esri — addresses this directly. Rather than treating splats as an accidental point cloud, the extension defines them as a native graphics primitive inside the glTF container — the same format widely adopted for real-time 3D asset delivery. A compliant file can carry cameras, transforms, and animations alongside the splat data. For viewers that do not yet support the extension, the file degrades gracefully to a standard sparse point cloud rather than failing entirely.

The compression layer is where the practical impact lands. The SPZ format — backed by Niantic and the Khronos working group — compresses a raw PLY file by up to 90% with near-zero loss in visual fidelity, achieved by aggressively packing the Spherical Harmonics data. Qualcomm's competing L-GSC format takes a different approach: hardware-level compression designed for mobile chips and headsets, allowing compressed splats to be decoded natively on the GPU without a CPU decompression phase. The 3DTiles format (OGC) enables studios to stream massive volumetric environments over standard web connections for instant client previews — removing the requirement for local installation at the review stage.

The transition from PLY to standardised, compressed formats is what moves 3DGS from isolated multi-gigabyte data dumps to agile, streamable production assets — and what makes browser-based client previews and on-set pipeline management viable at scale.

The compression problem was being attacked from the moment the original paper landed. In September 2023, graphics programmer Aras Pranckevičius published early work on reducing splat storage using Morton-ordered chunking and GPU texture compression formats, achieving 5× compression at high quality before any formal standardisation effort existed. The community-driven work that followed directly informed the standardisation ecosystem that emerged in 2025–2026.

Streaming Splats via Cesium for Unreal

Standardising the format resolves the storage and metadata problem. A complementary development addresses the performance bottleneck in Unreal directly. Rather than loading a monolithic .ply file into hardware VRAM — where frame rate collapses under the weight of an unmanaged multi-gigabyte asset — the environment can be streamed through Cesium's geospatial architecture instead.

The pipeline runs through Cesium ion. A raw .ply file (or raw photographs processed via the integrated iTwin Capture engine) is uploaded to an ion account, where it is converted into a 3D Tileset using glTF 2.0 as the payload. The tiling process applies the KHR_gaussian_splatting extension — treating each splat as a mesh point primitive with custom ellipsoidal rendering rules — then applies Niantic's open-source SPZ compression (KHR_gaussian_splatting_compression_spz_2), reducing the storage footprint by up to 90% while preserving view-dependent specular reflections. The result is called from the standard Cesium for Unreal panel via asset ID and streams directly into the level viewport.

The practical advantages over traditional monolithic Unreal splat plugins are structural. Hierarchical Level of Detail (HLOD) means the engine delivers sub-centimetre visual fidelity where the virtual camera is pointed while down-sampling background clusters — the frame rate problem disappears. Georeferencing via the 3D Tiles index anchors the environment precisely to its real-world latitude, longitude, and altitude, placing it seamlessly alongside Cesium World Terrain or Google Photorealistic 3D Tiles. Renderer fallbacks are handled gracefully: because the data is structured inside standard glTF point primitives, platforms that do not support volumetric splat shaders render a clean sparse point cloud rather than failing the project.

Cesium's engineering team has published a detailed breakdown of the mechanics of embedding radiance fields into open formats and optimising them for large-scale streaming workloads — the Gaussian Splatting with 3D Tiles and glTF presentation covers the full technical architecture.

4D Gaussian Splatting

One branch of the research deserves particular attention. 4DGS extends the model by adding time as a fourth dimension — capturing not just where things are, but how they move. A static location scan becomes a navigable event. The implications for live performance, sports, and cultural preservation are significant enough to warrant a separate conversation, which is where this series goes next.

The Consumer Crossing

The clearest signal that 3DGS has moved from production tool to mainstream infrastructure arrived just days ago — not from a research paper or an industry pipeline announcement, but from two product updates most people encountered without knowing the technology behind them.

Google Maps quietly extended its Immersive View for Routes feature, using AI models trained on geotagged imagery at Flickr scale to generate fluid, first-person simulations of journeys before you take them. The photorealistic fly-throughs that users see as a navigation aid are built on the same scene reconstruction principles that VFX pipelines spent years developing.

At WWDC, Apple announced a comprehensive visual overhaul of Maps — described in marketing terms as a "fidelity boost" to Flyover driven by "radiance field representation." The technical description is 3DGS at scale, backed by SHARP, Apple's own open-source model for generating a Gaussian Splat from a single photograph in under a second — reconstructing depth and geometry within the captured field of view.

Neither company foregrounded the algorithm. Both shipped it as a product improvement. That is what mainstream adoption looks like.

What This Means for the Devices in Your Hands

Apple's deployment is not confined to Maps. The same underlying capability — SHARP's ability to reconstruct depth and geometry from a single photograph — has direct implications for Vision Pro. A flat iPhone photo becomes a spatially navigable environment. You do not view the memory. You step into it. The point of origin — the specific lighting, atmosphere, and time of day the image was captured — can now be separated from the geometry through relighting pipelines and adjusted independently. The photograph stops being a fixed record and becomes a space with variable conditions.

This is the direction spatial computing has been pointing toward for some time. What 3DGS provides is the rendering substrate that makes it work at the photorealistic threshold required. Standard 3D models look constructed in augmented reality. Gaussian Splats do not — they carry the light and material behaviour of the original capture in a way that polygon geometry cannot replicate.

Presence at Distance

Varjo Teleport approaches the same problem from a different direction. Rather than reconstructing environments, it captures people — using volumetric multi-camera rigs to record a human subject in three dimensions and stream that capture into an XR environment in real time. The result is photorealistic human presence across physical distance. Not a video call. Not an avatar. A volumetrically accurate representation of a person, present in your space.

For filmmaking, the implications are significant. A director based in London can be present on a set in New Zealand — not as a face on a monitor but as a spatial presence the crew can orient around. The distinction between being on location and being remote becomes a question of degree rather than kind.

Meta Hyperscapes

Meta's Hyperscapes project extends 3DGS into persistence. Rather than a single-user splat environment, Hyperscapes creates shared spatial layers — Gaussian Splat-rendered environments that overlay physical space and persist across sessions. Multiple users inhabit the same volumetric environment simultaneously. For pre-production, this means a director, production designer, and DP can walk a real location together as a splat — in different cities — before a single crew member travels.

A Lesson in How We Perceive Space

Ultimately, the pipeline breaking clean of the polygon is not a story about software architecture, rendering overhead, or file compression. It is a fundamental lesson in how the human brain perceives reality.

For decades, computer graphics forced the world into the rigid, artificial language of Euclidean geometry — demanding that space be defined by hard boundaries, wireframe meshes, and flat textures. But human vision has never operated that way. We do not navigate our daily lives by computing the absolute geometric coordinates of every surface we see. We perceive the universe as a continuous, direction-dependent tapestry of light transport, shifting reflections, and fuzzy volumetric depth.

By abandoning the mesh and storing reality as an explicit cloud of semi-transparent light fields, 3D Gaussian Splatting aligns the machine's representation of space with our own cognitive architecture. It works because it mirrors the exact optical hacks our eyes and visual cortex use to form the illusion of the world around us.

The fact that these exact same primitives are simultaneously driving consumer navigation in Google Maps, anchoring shared spatial persistence in MR headsets, and building cinematic worlds proves that this evolution spans far beyond any single industry. We are moving away from an era where we merely sliced history into flat, two-dimensional frames. The long-term legacy of this shift is that our relationship with distance, location, and human memory has permanently transformed. We are no longer just recording the world; we are archiving the very illusion of experiencing it.