Tele-Aloha: A Low-budget and High-authenticity Telepresence System Using Sparse RGB Cameras

SIGGRAPH 2024

Hanzhang Tu¹
Ruizhi Shao¹
Xue Dong²
Shunyuan Zheng^{3, †}
Hao Zhang²
Lili Chen²
Meili Wang²
Wenyu Li²
Siyan Ma²
Shengping Zhang³
Boyao Zhou^{1, *}
Yebin Liu^{1, *}

¹ Tsinghua University
² BOE Technology Group
³ Harbin Institute of Technology

* Corresponding authors. † Work done during an internship at Tsinghua University.

Telepresence
Real-time systems

Read paper Watch video

Telepresence illustration

TL;DR

Tele-Aloha is a bidirectional telepresence system for peer-to-peer communication. Sparse RGB capture, neural rendering, and an autostereoscopic display bring a remote person's upper body into a view-dependent, three-dimensional communication experience.

Abstract

In this paper, we present a low-budget and high-authenticity bidirectional telepresence system, Tele-Aloha, targeting peer-to-peer communication scenarios. Compared to previous systems, Tele-Aloha utilizes only four sparse RGB cameras, one consumer-grade GPU, and one autostereoscopic screen to achieve high-resolution (2048 × 2048), real-time (30 fps), low-latency (less than 150 ms) and robust distant communication.

As the core of Tele-Aloha, we propose an efficient novel view synthesis algorithm for upper-body. Firstly, we design a cascaded disparity estimator for obtaining a robust geometry cue. Additionally a neural rasterizer via Gaussian Splatting is introduced to project latent features onto target view and to decode them into a reduced resolution. Further, given the high-quality captured data, we leverage weighted blending mechanism to refine the decoded image into the final resolution of 2K.

Exploiting world-leading autostereoscopic display and low-latency iris tracking, users are able to experience a strong three-dimensional sense even without any wearable head-mounted display device. Altogether, our telepresence system demonstrates the sense of co-presence in real-life experiments, inspiring the next generation of communication.

4 RGB cameras: Sparse upper-body capture
2048 × 2048: Novel-view image resolution
30 fps: Real-time communication
1 RTX 4090: Consumer-grade GPU

Tele-Aloha capture and display setup, with four RGB cameras around an autostereoscopic screen showing a remote participant. — Tele-Aloha: A Low-budget and High-authenticity Telepresence System Using Sparse RGB Cameras.
Full-size figure

Comparison with previous telepresence systems

Scroll horizontally to see all columns. Keyboard: focus the table and use the arrow keys.

Published system configurations, as compared on the Tele-Aloha reference page.
System	Input setting	Efficiency	Resolution	GPU equipment	Complexity
Holoportation	8 RGBD	34 fps	—	10 Titan X	Medium (F.B.)
LookinGood	1 RGBD	34 fps	512 × 1024	1 Titan V	High (U.B.)
Starline	3 RGBD	60 fps	1600 × 1200	2 RTX 6000 + 2 Titan RTX	High (U.B.)
VirtualCube	6 RGBD	30 fps	1280 × 960	2 RTX 3090 + 1 RTX 2080	High (U.B.)
MetaStream	4 RGBD	30 fps	—	4 Jetson Nano + 1 RTX 2080S	Medium (F.B.)
AI-mediated	1 RGB	24 fps	512 × 512	1 RTX 6000 + 1 RTX 4090	Low (Head)
Live4d	20 RGB	24 fps	—	4 RTX 3090 + 1 RTX 4090	Medium (F.B.)
Tele-Aloha (ours)	4 RGB	30 fps	2048 × 2048	1 RTX 4090	High (U.B.)

F.B. = full body; U.B. = upper body; RGBD = color and depth. A dash denotes no directly comparable image resolution in the source table.

These systems use different hardware and capture settings. This is a comparison of published configurations, not a controlled benchmark on identical equipment.

Efficient novel view synthesis

A cascaded disparity estimator supplies geometry cues. Gaussian splatting projects latent features to the target view, and weighted blending refines the image. Reconstruction, streaming, and display form an end-to-end real-time pipeline.

Cascaded Disparity Estimation
The upper pair has a smaller baseline, which is more robust for stereo matching, while the lower pair has a larger field of view. Therefore, we propose to estimate the disparity of the upper pair first and adopt it as initialization to estimate the disparity of the lower pair with more stability.
Neural Rasterization via 3DGS
The image encoder predicts the Gaussian properties instantly. Then, the latent-based 3DGS is rendered as the latent map at the novel view, and the neural image decoder is utilized to recover RGB images.
Occlusion-aware Rendering Refinement
A lightweight refinement module is engaged to generate the final output in full resolution. To fully leverage the high-resolution input of the system, it takes the refined image feature and a high-resolution image blended from source views as input, to produce high-resolution images.

Tele-Aloha pipeline showing cascaded disparity estimation, Gaussian neural rasterization, and high-resolution rendering refinement. — The pipeline of the proposed novel view synthesis algorithm. Given four input views that form two stereo pairs, we first estimate the disparity maps in the perspectives of these input views in a cascaded manner. Then, together with feature maps, scale maps, and opacity maps extracted by an image encoder, the pixels are lifted to 3D to form pixel-aligned 3DGS points. After that, feature vectors are rasterized to selected novel views, followed by a decoder that turns feature maps back into RGB images. Finally, a refiner module is applied to upsample the novel view rendering.
Full-size figure

Realtime bidirectional communication

Realtime bidirectional communicationExternal video · MP4

Blur and ghosting are due to monitor characteristics: the autostereoscopic display tracks only the eyes of the current user.

Static results on the THuman dataset

The proposed method is trained on THuman-Sit and THuman2.0. We split the THuman-Sit scans into two parts, 4000 for training and 700 for testing. We render source images parallel to our hardware setup and render novel views randomly among them.

Two rows comparing seated people reconstructed by FloRen, ENeRF, GPS-Gaussian, and Tele-Aloha against ground truth; red insets enlarge clothing and arm details. — Comparison on sparse-view synthetic dataset against Floren, ENeRF, and GPS-gaussian.
Full-size figure

Demo video

Demo videoExternal video · MP4

The original Tele-Aloha demonstration, hosted externally. Use the player controls to start playback.

Project scope

The cover illustrates remote communication. See the paper for system hardware, evaluation, and limitations.

Citation

BibTeX

@inproceedings{tu2024telealoha,
  title = {Tele-Aloha: A Low-budget and High-authenticity
           Telepresence System Using Sparse RGB Cameras},
  author = {Tu, Hanzhang and Shao, Ruizhi and Dong, Xue and
            Zheng, Shunyuan and Zhang, Hao and Chen, Lili and
            Wang, Meili and Li, Wenyu and Ma, Siyan and
            Zhang, Shengping and Zhou, Boyao and Liu, Yebin},
  booktitle = {ACM SIGGRAPH 2024 Conference Papers},
  year = {2024},
  doi = {10.1145/3641519.3657491}
}