Download Video: HD (MP4, 101 MB)


We present the first real-time human performance capture approach that reconstructs dense, space-time coherent deforming geometry of entire humans in general everyday clothing from just a single RGB video.We propose a novel two-stage analysis-by-synthesis optimization whose formulation and implementation are designed for high performance. In the first stage, a skinned template model is jointly fitted to background subtracted input video, 2D and 3D skeleton joint positions found using a deep neural network, and a set of sparse facial landmark detections. In the second stage, dense non-rigid 3D deformations of skin and even loose apparel are captured based on a novel real-time capable algorithm for non-rigid tracking using dense photometric and silhouette constraints. Our novel energy formulation leverages automatically identified material regions on the template to model the differing non-rigid deformation behavior of skin and apparel. The two resulting nonlinear optimization problems per-frame are solved with specially-tailored data-parallel Gauss-Newton solvers. In order to achieve real-time performance of over 25Hz, we design a pipelined parallel architecture using the CPU and two commodity GPUs. Our method is the first real-time monocular approach for full-body performance capture. Our method yields comparable accuracy with off-line performance capture techniques, while being orders of magnitude faster.



BibTeX, 1 KB

 author = {Habermann, Marc and Xu, Weipeng and Zollh\"{o}fer, Michael and Pons-Moll, Gerard and Theobalt, Christian},
 title = {LiveCap: Real-Time Human Performance Capture From Monocular Video},
 journal = {ACM Trans. Graph.},
 issue_date = {April 2019},
 volume = {38},
 number = {2},
 month = mar,
 year = {2019},
 issn = {0730-0301},
 pages = {14:1--14:17},
 articleno = {14},
 numpages = {17},
 url = {},
 doi = {10.1145/3311970},
 acmid = {3311970},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {3D pose estimation, Monocular performance capture, human body, non-rigid surface deformation},


This work was funded by the ERC Consolidator Grant 4DRepLy (770784).


For questions, clarifications, please get in touch with:
Marc Habermann

This page is Zotero translator friendly. Page last updated Imprint. Data Protection.