Bypassing the Mesh III: The Operational 3DGS Pipeline from Scout to LED

Part of the Bypassing the Mesh series. The hub article covers the full landscape. This spoke traces an environment from physical location capture through to live LED wall deployment.

The problem 3D Gaussian Splatting solves is a specific one: how do you take a real physical environment and reproduce it as a high-quality, photorealistic virtual space that performs in real time? Not built from scratch. Captured from reality. The technology has officially crossed from research curiosity into a repeatable, stage-ready workflow. This playbook is for Technical Directors, Virtual Production Supervisors, and anyone building 3DGS pipelines across industries.

Phase 1: Hybrid Capture & Compliance Vetting

The physical capture phase establishes the ultimate quality ceiling of the digital twin; you cannot compute your way out of a sloppy capture. To prevent structural gaps and data drift, teams must deploy a hybrid, multi-layered capture strategy while strictly auditing hardware origin.

Aerial Passes: Systematic drone paths mapping the global structure and upper surfaces of the environment. A strict minimum of 80% image overlap is required.

Handheld Detail: Ground-level walking passes capturing interior geometry, tight corridors, and low-angle surfaces. Photogrammetry struggle-points are supplemented using LiDAR-fusion rigs (such as XGRIDS) or multi-camera arrays.

Reference Anchor: A known physical geometry or coordinate baseline (such as a quick terrestrial scan) that ties the aerial and handheld datasets together, completely preventing post-production dataset drift.

Manufacturer	Model	Best Used For	Compliance Status
Freefly Systems	Astro Prime	High-res photogrammetry (61MP Sony sensor)	Blue UAS Cleared / NDAA Compliant
Skydio	X10	High autonomy & rapid mapping in complex spaces	Blue UAS Cleared / NDAA Compliant
Skyfish	M4 / M6	Precise industrial mapping and massive digital twins	Blue UAS Cleared / Green UAS Certified
ACSL (Japan)	SOTEN	Secure, encrypted close-quarters mapping	NDAA Compliant

Phase 2: Data Processing, Reconstruction & De-lighting

Once raw images and spatial packets are extracted, they route into a dedicated processing and optimisation pipeline. Raw captures are plagued by "floaters" (orphaned, mid-air splat particles), transient background movement, and baked-in location illumination that will conflict with studio lights on set.

Studios balance this phase across two primary processing pathways: Hardware-Linked Turnkey Ecosystems or Open-Source Local Pipelines.

1. The Processing and Reconstruction Track

The XGRIDS Pipeline

For productions using XGRIDS capture hardware (such as the Lixel series), raw sensor data routes directly through their dedicated software suites based on the VAD deliverable requirement:

LCC Studio (3DGS Pipeline): The primary reconstruction engine that digests raw, synchronised sensor trajectories into render-ready environments. It handles one-click reconstruction into standard .ply or compressed .lcc/.lcc2 formats and features Air-Ground Map Fusion to align aerial drone data with ground LiDAR sweeps automatically. Large-scale environments can be exported directly as hierarchical 3D Tiles (SPZ v2) for seamless engine streaming.

LixelStudio (SLAM & Engineering Pipeline): Deployed when the required output is structural proxy geometry, survey data, or collision meshes. It runs automated SLAM mapping with loop closure and dynamic object removal (filtering out moving crew or vehicles), utilising PPK & RTK alignment to lock the twin to exact real-world geographical coordinates.

The Open-Source Local Track

Productions avoiding platform lock-in or third-party SaaS fees utilise automated open-source local processing stacks:

Jawset Postshot: A powerful local desktop solution that handles training entirely on local studio workstations, offering tight command-line control over clipping, pruning, and splat budget optimisation with native ACES colour space management.

Nerfstudio (Splatfacto): The global R&D pipeline standard. Allows technical directors to write entirely custom, scriptable ingestion pipelines from scratch.

Scaniverse (.spz): Used by VAD teams for rapid location scouting, using .spz compression to shrink multi-gigabyte files by up to 90% while retaining the layout fidelity needed for pre-production.

2. The De-lighting and Primitive Cleaning Challenge

Floater Removal

Before engine ingestion, the raw .ply point cloud is loaded into SuperSplat (an open-source, web-based visual editor). Editors manually inspect the cloud, using lasso tools to non-destructively purge erroneous noise, glass reflections, and stray artefacts.

Resolving Baked-In Light

3DGS natively captures light as it exists in the space. A raw splat contains all ambient and direct lighting present at the time of capture. Left unmanaged, changing the digital lights on an LED stage will conflict with the baked-in shadows and highlights of the asset.

The Volinga Pathway: Raw data is processed and compressed into the proprietary .nvol format. The Volinga Plugin Pro splits the volumetric data into distinct layers, leveraging proxy meshes to compute mesh-based relighting and shadows. This strips away static sunlight, delivering a neutral asset that the stage DP can light natively and dynamically.

The Sony Pathway: Sony's integrated XYN Spatial Capture uses proprietary inverse algorithms to isolate complex surface characteristics including gloss, material transparency, and atmospheric depth, generating photorealistic 3DCG assets directly from imagery without manual mesh reconstruction.

Inverse Rendering (IR) & RUGS: Open-source frameworks that mathematically decompose scene physics during training. They separate the asset into fixed appearance data and variable image-based lighting (IBL) components, allowing compute shaders to light the splat naturally without requiring a single proxy mesh.

Worth noting for productions on tight post-production schedules: inverse rendering demands significantly higher training compute than standard splat training, and this overhead needs to be factored into pipeline planning from the outset.

Geometry-Implicit VP Relighting: Bleeding-edge pipelines that sample the LED wall's background imagery directly in pixel space. Splats automatically calculate their own exposure and reflections based on whatever content is playing on the volume.

Phase 3: Real-Time Ingestion & Stage Playback

Bringing a fully trained, de-lighted 3DGS environment onto the LED stage requires a reliable real-time playback system. 3DGS utilises custom rasterisation pipelines rather than traditional polygon rendering, which means choice of playback platform directly alters production flexibility, performance overhead, and colour management.

The three primary engineering pathways for driving ICVFX volumes each present distinct operational trade-offs:

Playback System	Tracking & Camera Integration	Multi-Node Scaling	Colour Pipeline & System Overhead
Disguise (RenderStream)	Industry-standard camera tracking integration via hardware cluster abstraction. Interoperable with major tracking protocols out of the box.	Excellent. Manages cross-node synchronisation and viewport splitting across multiple render nodes automatically.	High-bandwidth video streaming overhead via RenderStream protocol, requiring dedicated network infrastructure. Full hardware colour management via internal software colour profiles.
Native Unreal Engine (nDisplay)	Deeply integrated into the inner frustum pipeline. Synchronised directly with engine-level camera components.	Configurable but complex. Requires meticulous manual setup of nDisplay cluster configuration files and network switches.	Minimal overhead. Renders directly to the GPU frame buffer without an intermediary streaming layer. Supports native ACES pipelines within the Unreal Engine environment.
Foundry Nuke Stage	Direct alignment with live tracking metadata within a traditional compositing interface, ideal for 2.5D or plate-heavy environments.	Limited. Best suited for single-node setups or simpler physical stage layouts; less adapted for massive, multi-display LED volumes.	Superior. Operates natively within Nuke's linear colour pipeline, allowing live, node-based colour grading and compositing on set.

Architectural Data Flow & Streaming

For massive exterior locations, loading a raw, multi-gigabyte splat directly into VRAM causes immediate engine failure. The production data flow relies on distributed streaming frameworks:

[Live Camera Tracking Feed]
           │
           ▼
[Disguise RenderStream] ─── (Synchronises Lens Data & Frustum Tracking)
           │
           ▼
[Unreal Engine 5 Core] ──── (Renders Volumetric 3DGS Asset via Custom Shaders)
           │
           ▼
[Cesium for Unreal] ─────── (Streams Kilometre-Wide Assets via 3D Tiles / HLOD)

Cesium for Unreal resolves VRAM bottlenecks by streaming data via 3D Tiles and dynamically managing memory limits through Hierarchical Levels of Detail (HLOD) based on what the lens actually sees. Disguise RenderStream then handles the heavy lifting, distributing the synchronised workload across the stage's nDisplay render nodes.

Bypassing the Niagara Bottleneck

Early engine plugins tried to force splats into Unreal Engine's Niagara Particle System, which crawls and chokes once you hit a few million points. Modern pipelines bypass Niagara completely using native, low-level GPU rasterisation:

MLSLabsRenderer: The technical director favourite for Unreal Engine 5. A custom low-level CUDA rendering pipeline that bypasses Niagara entirely, handling over 7 million Gaussian points at a stable 50+ FPS on consumer GPUs with perfect colour consistency against Unreal's post-processing. Natively supports 4DGS (Dynamic Volumetric Video) with full UE Sequencer integration.

UnityGaussianSplatting (Aras Pranckevičius): For studios running a Unity-based virtual production workflow, this open-source framework brings native, high-performance .ply playback directly into Unity's High Definition Render Pipeline (HDRP).

Phase 4: Pre-Visualisation & VR Scouting Upgrades

Traditional pre-visualisation (previs) relies on flat, grey-box geometry proxies that lack spatial or textural truth. The 3DGS pipeline collapses the gap between pre-production and principal photography. The splat running in previs is the exact asset that will run on the LED wall. Camera choreography, lens selections, and composition choices finalised in the production office hold perfectly when the physical stage goes live.

The VR Scouting Extension

To maximise the value of the 3DGS asset before stepping onto the physical stage, productions append a Virtual Scouting block directly onto the Previs phase. Using Unreal Engine's VR Scouting toolsets, the Director, DP, and Virtual Art Department (VAD) can put on VR headsets to step directly inside the 1:1 scale environment. This unlocks two critical capabilities:

True Spatial Blocking: Creative teams can walk the digital location months before the shoot, utilising VR controllers to drop virtual cameras, lock in storyboard heights, and test lens focal lengths in a 360° spatial context.

Stage Constraint Verification: Directors can visually evaluate real-world scale — such as checking if a virtual ceiling or structural beam will feel claustrophobic or sit too low relative to the physical height limits of the LED volume.

Hardware-Targeted Stage Validation

For productions utilising the Sony Virtual Production Tool Set, this previs state hooks directly into hardware-specific validation. The Camera and Display Plugin for Unreal Engine allows the VAD team to simulate a physical VENICE camera directly inside the digital volume, allowing the DP to forecast exact exposure thresholds, depth-of-field falloff, and identify potential moiré interference risks specific to the pixel pitch of Crystal LED B-series and C-series display configurations before a single crew member arrives on set. On shoot day, the Colour Calibrator app records a baseline chart through the camera to automate custom 3D LUT generation, aligning the LED wall and the sensor profile seamlessly.

Managing the "Expectation Gap"

The hub article covers the Expectation Gap in full. On set, three protocols manage it:

1. Viewport Split: Always show the crew the final, colour-graded camera output rather than the raw, un-composited background render feed. The camera's natural depth-of-field and lens properties mask edge degradation effectively.
2. Close-Up Boundaries: Establish the camera's spatial degradation zones during the Previs/VR Scouting phase. Document these limits and brief the physical camera team before shoot day, not on it.
3. Asset-as-Illusion Briefing: The creative leads must align on the fact that a 3DGS environment is a directional illusion with rigid spatial boundaries, not a physical environment that can be shot from any arbitrary angle without planning.