Franziska Mueller

Franziska Mueller

Max-Planck-Institut für Informatik
Department 4: Computer Graphics
 office: Campus E1 4, Room 211 C
Saarland Informatics Campus
66123 Saarbrücken


 email: Get my email address via email
 phone: +49 681 9325 4057
I'm a 5th year Ph.D. student in the Graphics, Vision and Video group supervised by Prof. Dr. Christian Theobalt.
I started as a Research Scientist at Google Zurich in October 2020. My new website will be online soon (check back here for the link).

Research Interests

  • Real-time tracking of interacting hands
  • Model- and optimization-based reconstruction of articulated motion
  • Machine learning techniques for computer vision


RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video

J. Wang, F. Mueller, F. Bernard, S. Sorli, O. Sotnychenko, N. Qian, M. A. Otaduy, D. Casas, C. Theobalt

ACM Transactions on Graphics (Proc. of SIGGRAPH Asia 2020)

In this work we present the first real-time method for motion capture of skeletal pose and 3D surface geometry of two hands from a single RGB camera that explicitly considers close interactions.

[project page] [pdf] [video]

HTML: A Parametric Hand Texture Model for 3D Hand Reconstruction and Personalization

N. Qian, J. Wang, F. Mueller, F. Bernard, V. Golyanik, C. Theobalt

European Conference on Computer Vision (ECCV) 2020

In this work we present HTML, the first parametric texture model of human hands. Our model spans several dimensions of hand appearance variability (e.g., related to gender, ethnicity, or age) and only requires a commodity camera for data acquisition. Experimentally, we demonstrate that our appearance model can be used to tackle a range of challenging problems such as 3D hand reconstruction from a single monocular image. Furthermore, our appearance model can be used to define a neural rendering layer that enables training with a self-upervised photometric loss. We make our model publicly available.

[project page] [pdf]

Generative Model-Based Loss to the Rescue: A Method to Overcome Annotation Errors for Depth-Based Hand Pose Estimation

J. Wang, F. Mueller, F. Bernard, C. Theobalt

International Conference on Automatic Face and Gesture Recognition (FG) 2020

In this paper we propose to use a model-based generative loss for training hand pose estimators on depth images based on a volumetric hand model. We demonstrate that such an approach can be used to train on datasets that have erroneous annotations while obtaining predictions that explain the depth images better than the annotions.

[project page] [pdf] [video]

XNect: Real-time Multi-person 3D Human Pose Estimation with a Single RGB Camera

D. Mehta, O. Sotnychenko, F. Mueller, W. Xu, M. Elgharib, P. Fua, H.P. Seidel, H. Rhodin, G. Pons-Moll, C. Theobalt

ACM Transactions on Graphics (Proc. of SIGGRAPH 2020)

A real-time multi-person 3D human body pose estimation system requiring only a single RGB camera for human motion capture in general scenes. The system produces full body 3D articulation estimates even under strong partial occlusion, as well as estimates of camera relative localization in space. The system can handle an arbitrary number of people in the scene, and processes complete frames without requiring prior person detection. A novel efficient convolutional network architecture (SelecSLS) enables real-time performance without compromising on accuracy. The pose formulation has the fully convolutional network only reason about the visible body parts, and occluded part reasoning is offloaded to a subsequent small fully connected network which gathers the full body context.

[arXiv] [Demo@CVPR19] [SelecSLS-Pytorch]

Convex Optimisation for Inverse Kinematics

T. Yenamandra, F. Bernard, J. Wang, F. Mueller, C. Theobalt

International Conference on 3D Vision (3DV) 2019

In this paper we propose a convex optimisation approach for the IK problem based on semidefinite programming, which admits a polynomial-time algorithm that globally solves (a relaxation of) the IK problem. Experimentally, we demonstrate that the proposed method significantly outperforms local optimisation methods using different real-world skeletons.


Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Camera

F. Mueller, M. Davis, F. Bernard, O. Sotnychenko, M. Verschoor, M. A. Otaduy, D. Casas, C. Theobalt

ACM Transactions on Graphics (Proc. of SIGGRAPH 2019)

We present a novel method for real-time pose and shape reconstruction of two strongly interacting hands. Our approach is the first two-hand tracking solution that combines an extensive list of favorable properties, namely it is marker-less, uses a single consumer-level depth camera, runs in real time, handles inter- and intra-hand collisions, and automatically adjusts to the user's hand shape.

[pdf] [video] [project page]

HandSeg: An Automatically Labeled Dataset for Hand Segmentation from Depth Images

A. Bojja, F. Mueller, S. Malireddi, M. Oberweger, V. Lepetit, C. Theobalt, K. Yi, A. Tagliasacchi

Conference on Computer and Robot Vision (CRV) 2019

We propose an automatic method for generating high-quality annotations for depth-based hand segmentation, and introduce a large-scale hand segmentation dataset

[arxiv] [project page]

FingerInput: Capturing Expressive Single-Hand Thumb-to-Finger Microgestures for On-Skin Input

M. Soliman, F. Mueller, L. Hegemann, J. Sol Roo, C. Theobalt, J. Steimle

ACM International Conference on Interactive Surfaces and Spaces (ISS) 2018
Best Academic Paper Award

In this paper, we present a consolidated design space for thumb-to-finger microgestures. Based on this design space, we introduce a thumb-to-finger gesture recognition system which achieves an average accuracy of 91% for real-time detection of 8 demanding gesture classes.

[pdf] [project page]

Single-Shot Multi-Person 3D Body Pose Estimation From Monocular RGB Input

D. Mehta, O. Sotnychenko, F. Mueller, W. Xu, S. Sridhar, G. Pons-Moll, C. Theobalt

International Conference on 3D Vision (3DV) 2018

A single-shot approach to jointly predict 3D body pose of multiple subjects in general scenes without requiring prior bounding-box extraction. Trained on new composited MuCo-3DHP dataset and evaluated on a new recorded multi-person 3D pose benchmark MuPoTS-3DHP.

[arxiv] [project page]

GANerated Hands for Real-Time 3D Hand Tracking from Monocular RGB

F. Mueller, F. Bernard, O. Sotnychenko, D. Mehta, S. Sridhar, D. Casas, C. Theobalt

Spotlight @ Computer Vision and Pattern Recognition (CVPR) 2018

We propose a novel method for real-time hand tracking from monocular RGB input which combines a CNN-based regressor and a kinematic optimization framework. To enhance our synthetic training data, we introduce a new geometrically consistent image-to-image translator for unpaired examples.

[pdf] [supplemental] [video] [spotlight presentation] [project page]

Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor

F. Mueller, D. Mehta, O. Sotnychenko, S. Sridhar, D. Casas, C. Theobalt

International Conference on Computer Vision (ICCV) 2017

We present a method for real-time hand tracking under occlusion in cluttered egocentric scenes from a single RGB-D camera. To enable training of our machine learning components, we introduce a new large-scale dataset SynthHands which was captured using a mixed reality approach. Furthermore, we propose a real benchmark dataset EgoDexter which provides annotated fingertip positions.

[pdf] [supplemental] [video] [project page]

Real-time Joint Tracking of a Hand Manipulating an Object from RGB-D Input

S. Sridhar, F. Mueller, M. Zollhöfer, D. Casas, A. Oulasvirta, C. Theobalt

European Conference on Computer Vision (ECCV) 2016

In this work we tackle the especially challenging problem of tracking joint hand and object motion while running at realtime framerates.

[pdf] [supplemental] [poster] [video] [project page] [Dexter+Object dataset]

Fast and Robust Hand Tracking Using Detection-Guided Optimization

S. Sridhar, F. Mueller, A. Oulasvirta, C. Theobalt

Computer Vision and Pattern Recognition (CVPR) 2015

We present a novel approach for real-time tracking of hand motion from a single commodity depth camera. The method combines a generative pose optimization framework with discriminative part label evidence for robustness and recovery.

[pdf] [supplemental] [poster] [video] [project page]

Awards & Honors


  • Summer semester 2019:
    Seminar Computer Vision and Machine Learning for Computer Graphics, Saarland University and MPI for Informatics
  • Summer semester 2018:
    Seminar 3D Shape Analysis, Saarland University and MPI for Informatics
  • Summer semester 2016, 2017:
    Seminar Computer Vision for Computer Graphics, Saarland University and MPI for Informatics
  • September/October 2015:
    Voluntary lecturer and coach for the Mathematik-Vorkurs für Informatiker (Math preparation course for new CS students), Saarland University
  • Winter semester 2014/15:
    Tutor for Grundzüge der Theoretischen Informatik (Theoretical Computer Science), Lecturer: Prof. Dr. Markus Bläser, Saarland University
  • Summer semester 2014:
    Tutor for Systemarchitektur (System Architecture), Lecturer: Prof. Dr. Jan Reineke, Saarland University
  • Winter semester 2013/14:
    Tutor for Programmierung 1 (Programming 1), Lecturer: Prof. Dr. Holger Hermanns, Saarland University
  • March 2013:
    Tutor for re-exam preparation in Programmierung 1 (Programming 1), Lecturer: Prof. Dr. Gert Smolka, Saarland University





  • Karnevalistischer Tanzsport (Carnival Dancing), a probably really German thing (with the awesome folks here or here on facebook). You can check out our most recent dance here.
  • Calisthenics
  • Reading (mostly fantasy)