• Real-time tracking of interacting hands
• Model- and optimization-based reconstruction of articulated motion
• Machine learning techniques for computer vision

RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video J. Wang, F. Mueller, F. Bernard, S. Sorli, O. Sotnychenko, N. Qian, M. A. Otaduy, D. Casas, C. Theobalt ACM Transactions on Graphics (Proc. of SIGGRAPH Asia 2020) In this work we present the first real-time method for motion capture of skeletal pose and 3D surface geometry of two hands from a single RGB camera that explicitly considers close interactions. [project page] [pdf] [video]

HTML: A Parametric Hand Texture Model for 3D Hand Reconstruction and Personalization N. Qian, J. Wang, F. Mueller, F. Bernard, V. Golyanik, C. Theobalt European Conference on Computer Vision (ECCV) 2020 In this work we present HTML, the first parametric texture model of human hands. Our model spans several dimensions of hand appearance variability (e.g., related to gender, ethnicity, or age) and only requires a commodity camera for data acquisition. Experimentally, we demonstrate that our appearance model can be used to tackle a range of challenging problems such as 3D hand reconstruction from a single monocular image. Furthermore, our appearance model can be used to define a neural rendering layer that enables training with a self-upervised photometric loss. We make our model publicly available. [project page] [pdf]

Generative Model-Based Loss to the Rescue: A Method to Overcome Annotation Errors for Depth-Based Hand Pose Estimation J. Wang, F. Mueller, F. Bernard, C. Theobalt International Conference on Automatic Face and Gesture Recognition (FG) 2020 In this paper we propose to use a model-based generative loss for training hand pose estimators on depth images based on a volumetric hand model. We demonstrate that such an approach can be used to train on datasets that have erroneous annotations while obtaining predictions that explain the depth images better than the annotions. [project page] [pdf] [video]

XNect: Real-time Multi-person 3D Human Pose Estimation with a Single RGB Camera D. Mehta, O. Sotnychenko, F. Mueller, W. Xu, M. Elgharib, P. Fua, H.P. Seidel, H. Rhodin, G. Pons-Moll, C. Theobalt ACM Transactions on Graphics (Proc. of SIGGRAPH 2020) A real-time multi-person 3D human body pose estimation system requiring only a single RGB camera for human motion capture in general scenes. The system produces full body 3D articulation estimates even under strong partial occlusion, as well as estimates of camera relative localization in space. The system can handle an arbitrary number of people in the scene, and processes complete frames without requiring prior person detection. A novel efficient convolutional network architecture (SelecSLS) enables real-time performance without compromising on accuracy. The pose formulation has the fully convolutional network only reason about the visible body parts, and occluded part reasoning is offloaded to a subsequent small fully connected network which gathers the full body context. [arXiv] [Demo@CVPR19] [SelecSLS-Pytorch]

Convex Optimisation for Inverse Kinematics T. Yenamandra, F. Bernard, J. Wang, F. Mueller, C. Theobalt International Conference on 3D Vision (3DV) 2019 In this paper we propose a convex optimisation approach for the IK problem based on semidefinite programming, which admits a polynomial-time algorithm that globally solves (a relaxation of) the IK problem. Experimentally, we demonstrate that the proposed method significantly outperforms local optimisation methods using different real-world skeletons. [pdf]

Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Camera F. Mueller, M. Davis, F. Bernard, O. Sotnychenko, M. Verschoor, M. A. Otaduy, D. Casas, C. Theobalt ACM Transactions on Graphics (Proc. of SIGGRAPH 2019) We present a novel method for real-time pose and shape reconstruction of two strongly interacting hands. Our approach is the first two-hand tracking solution that combines an extensive list of favorable properties, namely it is marker-less, uses a single consumer-level depth camera, runs in real time, handles inter- and intra-hand collisions, and automatically adjusts to the user's hand shape. [pdf] [video] [project page]

HandSeg: An Automatically Labeled Dataset for Hand Segmentation from Depth Images A. Bojja, F. Mueller, S. Malireddi, M. Oberweger, V. Lepetit, C. Theobalt, K. Yi, A. Tagliasacchi Conference on Computer and Robot Vision (CRV) 2019 We propose an automatic method for generating high-quality annotations for depth-based hand segmentation, and introduce a large-scale hand segmentation dataset [arxiv] [project page]

FingerInput: Capturing Expressive Single-Hand Thumb-to-Finger Microgestures for On-Skin Input M. Soliman, F. Mueller, L. Hegemann, J. Sol Roo, C. Theobalt, J. Steimle ACM International Conference on Interactive Surfaces and Spaces (ISS) 2018 Best Academic Paper Award In this paper, we present a consolidated design space for thumb-to-finger microgestures. Based on this design space, we introduce a thumb-to-finger gesture recognition system which achieves an average accuracy of 91% for real-time detection of 8 demanding gesture classes. [pdf] [project page]

Single-Shot Multi-Person 3D Body Pose Estimation From Monocular RGB Input D. Mehta, O. Sotnychenko, F. Mueller, W. Xu, S. Sridhar, G. Pons-Moll, C. Theobalt International Conference on 3D Vision (3DV) 2018 A single-shot approach to jointly predict 3D body pose of multiple subjects in general scenes without requiring prior bounding-box extraction. Trained on new composited MuCo-3DHP dataset and evaluated on a new recorded multi-person 3D pose benchmark MuPoTS-3DHP. [arxiv] [project page]

GANerated Hands for Real-Time 3D Hand Tracking from Monocular RGB F. Mueller, F. Bernard, O. Sotnychenko, D. Mehta, S. Sridhar, D. Casas, C. Theobalt Spotlight @ Computer Vision and Pattern Recognition (CVPR) 2018 We propose a novel method for real-time hand tracking from monocular RGB input which combines a CNN-based regressor and a kinematic optimization framework. To enhance our synthetic training data, we introduce a new geometrically consistent image-to-image translator for unpaired examples. [pdf] [supplemental] [video] [spotlight presentation] [project page]

Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor F. Mueller, D. Mehta, O. Sotnychenko, S. Sridhar, D. Casas, C. Theobalt International Conference on Computer Vision (ICCV) 2017 We present a method for real-time hand tracking under occlusion in cluttered egocentric scenes from a single RGB-D camera. To enable training of our machine learning components, we introduce a new large-scale dataset SynthHands which was captured using a mixed reality approach. Furthermore, we propose a real benchmark dataset EgoDexter which provides annotated fingertip positions. [pdf] [supplemental] [video] [project page]

Real-time Joint Tracking of a Hand Manipulating an Object from RGB-D Input S. Sridhar, F. Mueller, M. Zollhöfer, D. Casas, A. Oulasvirta, C. Theobalt European Conference on Computer Vision (ECCV) 2016 In this work we tackle the especially challenging problem of tracking joint hand and object motion while running at realtime framerates. [pdf] [supplemental] [poster] [video] [project page] [Dexter+Object dataset]

Fast and Robust Hand Tracking Using Detection-Guided Optimization S. Sridhar, F. Mueller, A. Oulasvirta, C. Theobalt Computer Vision and Pattern Recognition (CVPR) 2015 We present a novel approach for real-time tracking of hand motion from a single commodity depth camera. The method combines a generative pose optimization framework with discriminative part label evidence for robustness and recovery. [pdf] [supplemental] [poster] [video] [project page]

• November 2018:
  Best Academic Paper Award at the ACM International Conference on Interactive Surfaces and Spaces (ISS) 2018
• October 2017:
  Best Poster Award at the 3rd Workshop on Observing and Understanding Hands in Action (HANDS 2017)
• April 2017:
  Google PhD Fellowship 2017 (in Machine Perception, Speech Technology and Computer Vision)
• February 2017:
  Frauen-MINT-Award 2016
• December 2016:
  Award from the Erich-F.-Bläse Foundation for Research and Science
• November 2016:
  Günter-Hotz-Medal (award for the best Master graduates in CS)
• May 2015:
  Bachelor Award (for the best Bachelor graduates in CS)
• October 2013 - March 2016:
  German Academic Scholarship Foundation (Studienstiftung des Deutschen Volkes) scholarship
• April 2013 - February 2015:
  Member of the Bachelor Honors Program (special support program for talented and ambitious Bachelor students in CS)
• October 2012 - September 2013:
  Deutschlandstipendium scholarship
• June 2012:
  Völklinger Abiturpreis (award for the best high school diploma in the city of Völklingen)

• Summer semester 2019:
  Seminar Computer Vision and Machine Learning for Computer Graphics, Saarland University and MPI for Informatics
• Summer semester 2018:
  Seminar 3D Shape Analysis, Saarland University and MPI for Informatics
• Summer semester 2016, 2017:
  Seminar Computer Vision for Computer Graphics, Saarland University and MPI for Informatics
• September/October 2015:
  Voluntary lecturer and coach for the Mathematik-Vorkurs für Informatiker (Math preparation course for new CS students), Saarland University
• Winter semester 2014/15:
  Tutor for Grundzüge der Theoretischen Informatik (Theoretical Computer Science), Lecturer: Prof. Dr. Markus Bläser, Saarland University
• Summer semester 2014:
  Tutor for Systemarchitektur (System Architecture), Lecturer: Prof. Dr. Jan Reineke, Saarland University
• Winter semester 2013/14:
  Tutor for Programmierung 1 (Programming 1), Lecturer: Prof. Dr. Holger Hermanns, Saarland University
• March 2013:
  Tutor for re-exam preparation in Programmierung 1 (Programming 1), Lecturer: Prof. Dr. Gert Smolka, Saarland University

• July 2019 - November 2019:
  Research Intern at Facebook Reality Labs in Pittsburgh with Yaser Sheikh
• June 2015 - October 2015:
  Student Research Assistant at the Max-Planck-Institut für Informatik with Prof. Dr. Christian Theobalt

• June 2018 - September 2018:
  Research Visit at Stanford University hosted by Prof. Leonidas Guibas, Geometric Computation Group
• April 2016 - present:
  Ph.D. student in Computer Science supervised by Prof. Dr. Christian Theobalt at Saarland University, Saarbrücken, Germany and the Max-Planck-Institut für Informatik
  
• March 2015 - present:
  Member of the Graduate School of Computer Science at Saarland University, Saarbrücken, Germany
  
• April 2015 - March 2016:
  Master Studies in Computer Science at Saarland University, Saarbrücken, Germany
  Title of Master's Thesis: Real-Time Hand-Object Tracking Using a Single Depth Camera (supervisor: Prof. Dr. Christian Theobalt) [pdf]
• October 2012 - March 2015:
  Bachelor Studies in Computer Science at Saarland University, Saarbrücken, Germany
  Title of Bachelor's Thesis: Real-Time Hand Tracking Using Hybrid Pose Optimization (supervisor: Prof. Dr. Christian Theobalt) [pdf]
• June 2012:
  Abitur at the Warndtgymnasium Völklingen, Germany

Hobbies

• Karnevalistischer Tanzsport (Carnival Dancing), a probably really German thing (with the awesome folks here or here on facebook). You can check out our most recent dance here.
• Calisthenics
• Reading (mostly fantasy)