Tele360: Real-Time Feed-Forward Human Reconstruction from Sparse Unposed Cameras

ACM TOG · SIGGRAPH Asia 2026Accepted

Hanzhang Tu, Z. Liao, W. Min, J. Zhang, Y. Liu

3D vision
Telepresence

Telepresence illustration

TL;DR

Tele360 reconstructs dynamic humans from four to six sparse, unposed RGB cameras. The feed-forward pipeline jointly estimates camera poses and reconstructs dynamic 3D Gaussians, connecting live capture with free-viewpoint telepresence.

Abstract

Live free-viewpoint visualization of real humans is critical for immersive communication and interactive digital experiences. Existing methods either rely on computationally expensive optimization or require calibrated cameras and low-resolution inputs, making real-time high-resolution deployment impractical. In this work, we present Tele360, the first real-time feed-forward system for dynamic human reconstruction and live free-viewpoint visualization from sparse, unposed RGB streams.

Our system jointly estimates camera poses and reconstructs a dynamic 3D Gaussian representation for each time instance in a single forward pass. To achieve this, we start by designing a lightweight sparsity-aware multi-view transformer backbone that tokenizes foreground human regions while preserving global context through a shared scene token. We then employ a fully transformer-based Gaussian decoder to mitigate convolution-induced over-smoothing while keeping decoding sparse and efficient. In addition, we introduce a hybrid feature pyramid that injects multi-scale appearance cues into geometry prediction. We further introduce a lightweight differentiable Levenberg-Marquardt camera refinement layer to enhance multi-view consistency and geometric alignment.

Moreover, to stabilize learning under sparse, unposed inputs, we transfer multi-view geometry priors from large visual-geometry foundation model via teacher-student distillation. Finally, the predicted gaussian maps are streamed with video codecs to remote devices for interactive free-viewpoint rendering. Together, these designs form a balanced pipeline between efficiency and fidelity. Extensive experiments show that Tele360 achieves state-of-the-art visual quality while supporting 2K input-to-rendering in real time at over 25 FPS on a single consumer GPU across both studio benchmarks and challenging in-the-wild scenarios.

4–6 RGB cameras: Sparse, unposed input streams
2048 × 2048: 2K input-to-rendering
>25 fps: Reported input-to-rendering system
35.36 ms: Model inference · 6 views · RTX 5090

Tele360 system diagram showing unposed RGB capture, pixel-aligned Gaussian maps encoded into H.265 streams, and remote playback on handheld and immersive displays. — Fig. 1. We present Tele360, a human telepresence system using only 4–6 unposed RGB cameras, enabling real-time 3D reconstruction, streaming and interactive viewing of dynamic human gaussian at 2K resolution across multiple endpoints.
Full-size figure

Quantitative comparison of novel view synthesis

Scroll horizontally to see all columns. Keyboard: focus the table and use the arrow keys.

Table 1. Quantitative comparison of novel view synthesis on DNA-Rendering dataset.
Method	Calibrated input	Views	Alignment	PSNR ↑	SSIM ↑	LPIPS ↓	Time ↓
NoPoSplat	No	6	Yes	22.1677	0.8574	0.1433	~600 ms *
AnySplat	No	6	Yes	24.9883	0.9163	0.0781	~150 ms
DepthAnything3	No	6	No	21.1142	0.9115	0.0926	~300 ms
DoubleField	Yes	6	No	29.4737	0.9596	0.0523	≫1 s
GHG	Yes	6	No	26.2712	0.9458	0.0534	~150 ms
GPS-Gaussian	Yes	8 †	No	22.4023	0.8675	0.1312	~300 ms *
NeuS2	Yes	6	No	30.6106	0.9683	0.0336	≫1 s
Ours	No	6	No	32.1502	0.9689	0.0286	~35 ms

Calibrated baselines receive ground-truth camera poses. Lower-resolution outputs are upsampled for the 2K evaluation.

For two-view methods, adjacent views are paired as inputs, and timing is accumulated over all pairs (*). GPS-Gaussian requires stereo rectification, which may fail given 6 views (†). “Align” denotes an evaluation-time camera alignment step used for NoPoSplat and AnySplat when reporting novel-view synthesis metrics.

The precise 35.36 ms figure is model inference for six 2K views on an RTX 5090 (Table 6). It excludes capture, streaming, and remote display; the reported system frame rate is a separate measurement.

Architecture

Pixel-aligned Gaussian maps provide a structured interface between reconstruction and transmission through standard video codecs. The system brings together camera estimation, dynamic scene representation, and remote rendering.

Hybrid Feature Extraction
To exploit high-resolution visual cues without incurring excessive computation, we adopt a hybrid ViT–CNN design as shown in Fig. 2. Specifically, we keep the DINO feature branch at low resolution to respect the compute budget, and introduce a lightweight yet effective CNN to process the 2K inputs and produce a multi-scale feature pyramid that preserves fine-grained appearance details.
Sparsity-Aware Backbone
In human-centric captures, informative content is largely concentrated in the foreground, so applying dense ViT attention over full images wastes computation on background regions and hinders real-time deployment. We therefore adopt a sparsity-aware multi-view transformer backbone that restricts attention to foreground tokens while preserving the global context needed for stable camera estimation via a shared scene token.
Lightweight Gaussian Decoder
Building on this compact backbone, we adopt a fully transformer-based decoder to keep decoding sparse and efficient, while also mitigating the inherent low-pass behavior of convolutional DPT-style decoders that tends to oversmooth sharp depth discontinuities and introduce flying points near boundaries.

Tele360 architecture with hybrid DINO and CNN features, sparse foreground attention, camera estimation and refinement, and a lightweight Gaussian decoder. — Fig. 2. Pipeline overview. Given 4–6 uncalibrated RGB videos, we extract features with a frozen DINO and a lightweight CNN encoder, and predict a foreground mask to select foreground tokens. The hybrid features are fed into our sparsity-aware backbone and lightweight decoder to predict camera, depth maps, and Gaussian parameter maps. After the BA module, the final point maps used for Gaussian positions are obtained from the predicted depth maps and the refined camera poses.
Full-size figure

Qualitative comparisons

Nine rows compare NoPoSplat, AnySplat, DepthAnything3, DoubleField, GHG, GPS-Gaussian, NeuS2, Tele360, and ground truth across six clothed people, with enlarged face and garment details. — Fig. 6. Qualitative comparisons. We compare our method with state-of-the-art baselines. Zoom-in patches highlight fine-grained details. While existing methods suffer from blurry textures or severe geometric artifacts, our method successfully reconstructs high-fidelity details (e.g., facial features and footwear), closely matching the ground truth.
Full-size figure

Demo video

Demo videoExternal video · MP4

Tele360 demonstration.

Project scope

Accepted to ACM Transactions on Graphics (SIGGRAPH Asia 2026 Journal Track), as recorded in the supplied CV. The illustration is schematic, not a captured reconstruction.

Citation

ACM TOG · SIGGRAPH Asia 2026 · Accepted. The final citation and public paper link will be added when available.