Header logo is


2022


Reconstructing Expressive {3D} Humans from {RGB} Images
Reconstructing Expressive 3D Humans from RGB Images

Choutas, V.

ETH Zurich, Max Planck Institute for Intelligent Systems and ETH Zurich, December 2022 (thesis)

Abstract
To interact with our environment, we need to adapt our body posture and grasp objects with our hands. During a conversation our facial expressions and hand gestures convey important non-verbal cues about our emotional state and intentions towards our fellow speakers. Thus, modeling and capturing 3D full-body shape and pose, hand articulation and facial expressions are necessary to create realistic human avatars for augmented and virtual reality. This is a complex task, due to the large number of degrees of freedom for articulation, body shape variance, occlusions from objects and self-occlusions from body parts, e.g. crossing our hands, and subject appearance. The community has thus far relied on expensive and cumbersome equipment, such as multi-view cameras or motion capture markers, to capture the 3D human body. While this approach is effective, it is limited to a small number of subjects and indoor scenarios. Using monocular RGB cameras would greatly simplify the avatar creation process, thanks to their lower cost and ease of use. These advantages come at a price though, since RGB capture methods need to deal with occlusions, perspective ambiguity and large variations in subject appearance, in addition to all the challenges posed by full-body capture. In an attempt to simplify the problem, researchers generally adopt a divide-and-conquer strategy, estimating the body, face and hands with distinct methods using part-specific datasets and benchmarks. However, the hands and face constrain the body and vice-versa, e.g. the position of the wrist depends on the elbow, shoulder, etc.; the divide-and-conquer approach can not utilize this constraint. In this thesis, we aim to reconstruct the full 3D human body, using only readily accessible monocular RGB images. In a first step, we introduce a parametric 3D body model, called SMPL-X, that can represent full-body shape and pose, hand articulation and facial expression. Next, we present an iterative optimization method, named SMPLify-X, that fits SMPL-X to 2D image keypoints. While SMPLify-X can produce plausible results if the 2D observations are sufficiently reliable, it is slow and susceptible to initialization. To overcome these limitations, we introduce ExPose, a neural network regressor, that predicts SMPL-X parameters from an image using body-driven attention, i.e. by zooming in on the hands and face, after predicting the body. From the zoomed-in part images, dedicated part networks predict the hand and face parameters. ExPose combines the independent body, hand, and face estimates by trusting them equally. This approach though does not fully exploit the correlation between parts and fails in the presence of challenges such as occlusion or motion blur. Thus, we need a better mechanism to aggregate information from the full body and part images. PIXIE uses neural networks called moderators that learn to fuse information from these two image sets before predicting the final part parameters. Overall, the addition of the hands and face leads to noticeably more natural and expressive reconstructions. Creating high fidelity avatars from RGB images requires accurate estimation of 3D body shape. Although existing methods are effective at predicting body pose, they struggle with body shape. We identify the lack of proper training data as the cause. To overcome this obstacle, we propose to collect internet images from fashion models websites, together with anthropometric measurements. At the same time, we ask human annotators to rate images and meshes according to a pre-defined set of linguistic attributes. We then define mappings between measurements, linguistic shape attributes and 3D body shape. Equipped with these mappings, we train a neural network regressor, SHAPY, that predicts accurate 3D body shapes from a single RGB image. We observe that existing 3D shape benchmarks lack subject variety and/or ground-truth shape. Thus, we introduce a new benchmark, Human Bodies in the Wild (HBW), which contains images of humans and their corresponding 3D ground-truth body shape. SHAPY shows how we can overcome the lack of in-the-wild images with 3D shape annotations through easy-to-obtain anthropometric measurements and linguistic shape attributes. Regressors that estimate 3D model parameters are robust and accurate, but often fail to tightly fit the observations. Optimization-based approaches tightly fit the data, by minimizing an energy function composed of a data term that penalizes deviations from the observations and priors that encode our knowledge of the problem. Finding the balance between these terms and implementing a performant version of the solver is a time-consuming and non-trivial task. Machine-learned continuous optimizers combine the benefits of both regression and optimization approaches. They learn the priors directly from data, avoiding the need for hand-crafted heuristics and loss term balancing, and benefit from optimized neural network frameworks for fast inference. Inspired from the classic Levenberg-Marquardt algorithm, we propose a neural optimizer that outperforms classic optimization, regression and hybrid optimization-regression approaches. Our proposed update rule uses a weighted combination of gradient descent and a network-predicted update. To show the versatility of the proposed method, we apply it on three other problems, namely full body estimation from (i) 2D keypoints, (ii) head and hand location from a head-mounted device and (iii) face tracking from dense 2D landmarks. Our method can easily be applied to new model fitting problems and offers a competitive alternative to well-tuned traditional model fitting pipelines, both in terms of accuracy and speed. To summarize, we propose a new and richer representation of the human body, SMPL-X, that is able to jointly model the 3D human body pose and shape, facial expressions and hand articulation. We propose methods, SMPLify-X, ExPose and PIXIE that estimate SMPL-X parameters from monocular RGB images, progressively improving the accuracy and realism of the predictions. To further improve reconstruction fidelity, we demonstrate how we can use easy-to-collect internet data and human annotations to overcome the lack of 3D shape data and train a model, SHAPY, that predicts accurate 3D body shape from a single RGB image. Finally, we propose a flexible learnable update rule for parametric human model fitting that outperforms both classic optimization and neural network approaches. This approach is easily applicable to a variety of problems, unlocking new applications in AR/VR scenarios.

ps

pdf [BibTex]

2022


pdf [BibTex]


no image
Causality, causal digital twins, and their applications

Schölkopf, B.

Machine Learning for Science: Bridging Data-Driven and Mechanistic Modelling (Dagstuhl Seminar 22382), (Editors: Berens, Philipp and Cranmer, Kyle and Lawrence, Neil D. and von Luxburg, Ulrike and Montgomery, Jessica), September 2022 (talk)

ei

link (url) DOI [BibTex]

link (url) DOI [BibTex]


no image
Proceedings of the First Conference on Causal Learning and Reasoning (CLeaR 2022)

Schölkopf, B., Uhler, C., Zhang, K.

177, Proceedings of Machine Learning Research, PMLR, April 2022 (proceedings)

ei

link (url) [BibTex]

link (url) [BibTex]

2021


no image
Physically Plausible Tracking & Reconstruction of Dynamic Objects

Strecke, M., Stückler, J.

KIT Science Week Scientific Conference & DGR-Days 2021, October 2021 (talk)

ev

[BibTex]

2021


[BibTex]


no image
Proceedings of the 1st Workshop on NLP for Positive Impact

Field, A., Prabhumoye, S., Sap, M., Jin, Z., Zhao, J., Brockett, C.

Association for Computational Linguistics, August 2021 (proceedings)

ei

link (url) [BibTex]

link (url) [BibTex]


Improving Human Decision-Making by Discovering Efficient Strategies for Hierarchical Planning
Improving Human Decision-Making by Discovering Efficient Strategies for Hierarchical Planning

Heindrich, L., Consul, S., Stojcheski, J., Lieder, F.

Tübingen, Germany, The first edition of Life Improvement Science Conference, June 2021 (talk) Accepted

Abstract
The discovery of decision strategies is an essential part of creating effective cognitive tutors that teach planning and decision-making skills to humans. In the context of bounded rationality, this requires weighing the benefits of different planning operations compared to their computational costs. For small decision problems, it has already been shown that near-optimal decision strategies can be discovered automatically and that the discovered strategies can be taught to humans to increase their performance. Unfortunately, these near-optimal strategy discovery algorithms have not been able to scale well to larger problems due to their computational complexity. In this talk, we will present recent work at the Rationality Enhancement Group to overcome the computational bottleneck of existing strategy discovery algorithms. Our approach makes use of the hierarchical structure of human behavior by decomposing sequential decision problems into two sub-problems: setting a goal and planning how to achieve it. An additional metacontroller component is introduced to switch the current goal when it becomes beneficial. The hierarchical decomposition enables us to discover near-optimal strategies for human planning in larger and more complex tasks than previously possible. We then show in online experiments that teaching the discovered strategies to humans improves their performance in complex sequential decision-making tasks.

re

Project Page [BibTex]

Project Page [BibTex]

2020


no image
Voltage dependent interfacial magnetism in multilayer systems

Nacke, R.

Universität Stuttgart, Stuttgart, December 2020 (thesis)

mms

[BibTex]

2020


[BibTex]


Excursion Search for Constrained Bayesian Optimization under a Limited Budget of Failures
Excursion Search for Constrained Bayesian Optimization under a Limited Budget of Failures

Marco, A., Rohr, A. V., Baumann, D., Hernández-Lobato, J. M., Trimpe, S.

2020 (proceedings) In revision

Abstract
When learning to ride a bike, a child falls down a number of times before achieving the first success. As falling down usually has only mild consequences, it can be seen as a tolerable failure in exchange for a faster learning process, as it provides rich information about an undesired behavior. In the context of Bayesian optimization under unknown constraints (BOC), typical strategies for safe learning explore conservatively and avoid failures by all means. On the other side of the spectrum, non conservative BOC algorithms that allow failing may fail an unbounded number of times before reaching the optimum. In this work, we propose a novel decision maker grounded in control theory that controls the amount of risk we allow in the search as a function of a given budget of failures. Empirical validation shows that our algorithm uses the failures budget more efficiently in a variety of optimization experiments, and generally achieves lower regret, than state-of-the-art methods. In addition, we propose an original algorithm for unconstrained Bayesian optimization inspired by the notion of excursion sets in stochastic processes, upon which the failures-aware algorithm is built.

am ics

arXiv code (python) PDF [BibTex]

2019


no image
Multivariate coupling estimation between continuous signals and point processes

Safavi, S., Logothetis, N., Besserve, M.

Neural Information Processing Systems 2019 - Workshop on Learning with Temporal Point Processes, December 2019 (talk)

ei

Talk video link (url) [BibTex]

2019


Talk video link (url) [BibTex]

2018


Impact of Trunk Orientation  for Dynamic Bipedal Locomotion
Impact of Trunk Orientation for Dynamic Bipedal Locomotion

Drama, Ö.

Dynamic Walking Conference, May 2018 (talk)

Abstract
Impact of trunk orientation for dynamic bipedal locomotion My research revolves around investigating the functional demands of bipedal running, with focus on stabilizing trunk orientation. When we think about postural stability, there are two critical questions we need to answer: What are the necessary and sufficient conditions to achieve and maintain trunk stability? I am concentrating on how morphology affects control strategies in achieving trunk stability. In particular, I denote the trunk pitch as the predominant morphology parameter and explore the requirements it imposes on a chosen control strategy. To analyze this, I use a spring loaded inverted pendulum model extended with a rigid trunk, which is actuated by a hip motor. The challenge for the controller design here is to have a single hip actuator to achieve two coupled tasks of moving the legs to generate motion and stabilizing the trunk. I enforce orthograde and pronograde postures and aim to identify the effect of these trunk orientations on the hip torque and ground reaction profiles for different control strategies.

dlg

Impact of trunk orientation for dynamic bipedal locomotion [DW 2018] link (url) [BibTex]

2016


no image
Proceedings of the 32nd Conference on Uncertainty in Artificial Intelligence (UAI)

Ihler, A. T., Janzing, D.

pages: 869 pages, AUAI Press, June 2016 (proceedings)

ei

link (url) [BibTex]

2016


link (url) [BibTex]

2015


Proceedings of the 37th German Conference on Pattern Recognition
Proceedings of the 37th German Conference on Pattern Recognition

Gall, J., Gehler, P., Leibe, B.

Springer, German Conference on Pattern Recognition, October 2015 (proceedings)

ps

GCPR conference website [BibTex]

2015


GCPR conference website [BibTex]


no image
Causal Inference for Empirical Time Series Based on the Postulate of Independence of Cause and Mechanism

Besserve, M.

53rd Annual Allerton Conference on Communication, Control, and Computing, September 2015 (talk)

ei

[BibTex]

[BibTex]


no image
Independence of cause and mechanism in brain networks

Besserve, M.

DALI workshop on Networks: Processes and Causality, April 2015 (talk)

ei

[BibTex]

[BibTex]


no image
Policy Search for Imitation Learning

Doerr, A.

University of Stuttgart, January 2015 (thesis)

am ics

link (url) Project Page [BibTex]

link (url) Project Page [BibTex]


no image
Information-Theoretic Implications of Classical and Quantum Causal Structures

Chaves, R., Majenz, C., Luft, L., Maciel, T., Janzing, D., Schölkopf, B., Gross, D.

18th Conference on Quantum Information Processing (QIP), 2015 (talk)

ei

Web link (url) [BibTex]

Web link (url) [BibTex]


no image
The search for single exoplanet transits in the Kepler light curves

Foreman-Mackey, D., Hogg, D. W., Schölkopf, B.

IAU General Assembly, 22, pages: 2258352, 2015 (talk)

ei

link (url) [BibTex]

link (url) [BibTex]


no image
Unsupervised identification of neural events in local field potentials

Besserve, M., Schölkopf, B., Logothetis, N. K.

44th Annual Meeting of the Society for Neuroscience (Neuroscience), 2014 (talk)

ei

[BibTex]

[BibTex]


no image
Quantifying statistical dependency

Besserve, M.

Research Network on Learning Systems Summer School, 2014 (talk)

ei

[BibTex]

[BibTex]


no image
Development of advanced methods for improving astronomical images

Schmeißer, N.

Eberhard Karls Universität Tübingen, Germany, Eberhard Karls Universität Tübingen, Germany, 2014 (diplomathesis)

ei

[BibTex]

[BibTex]

2013


no image
Camera-specific Image Denoising

Schober, M.

Eberhard Karls Universität Tübingen, Germany, October 2013 (diplomathesis)

ei pn

PDF [BibTex]

2013


PDF [BibTex]


no image
Studying large-scale brain networks: electrical stimulation and neural-event-triggered fMRI

Logothetis, N., Eschenko, O., Murayama, Y., Augath, M., Steudel, T., Evrard, H., Besserve, M., Oeltermann, A.

Twenty-Second Annual Computational Neuroscience Meeting (CNS*2013), July 2013, journal = {BMC Neuroscience}, year = {2013}, month = {7}, volume = {14}, number = {Supplement 1}, pages = {A1}, (talk)

ei

Web [BibTex]

Web [BibTex]


no image
Proceedings of the 10th European Workshop on Reinforcement Learning, Volume 24

Deisenroth, M., Szepesvári, C., Peters, J.

pages: 173, JMLR, European Workshop On Reinforcement Learning, EWRL, 2013 (proceedings)

ei

Web [BibTex]

Web [BibTex]


no image
Domain Generalization via Invariant Feature Representation

Muandet, K.

30th International Conference on Machine Learning (ICML2013), 2013 (talk)

ei

PDF [BibTex]

PDF [BibTex]

2012


no image
Support Vector Machines, Support Measure Machines, and Quasar Target Selection

Muandet, K.

Center for Cosmology and Particle Physics (CCPP), New York University, December 2012 (talk)

ei

[BibTex]

2012


[BibTex]


no image
Hilbert Space Embedding for Dirichlet Process Mixtures

Muandet, K.

NIPS Workshop on Confluence between Kernel Methods and Graphical Models, December 2012 (talk)

ei

[BibTex]

[BibTex]


no image
Simultaneous small animal PET/MR in activated and resting state reveals multiple brain networks

Wehrl, H., Lankes, K., Hossain, M., Bezrukov, I., Liu, C., Martirosian, P., Schick, F., Pichler, B.

20th Annual Meeting and Exhibition of the International Society for Magnetic Resonance in Medicine (ISMRM), May 2012 (talk)

ei

Web [BibTex]

Web [BibTex]


no image
A new PET insert for simultaneous PET/MR small animal imaging

Wehrl, H., Lankes, K., Hossain, M., Bezrukov, I., Liu, C., Martirosian, P., Reischl, G., Schick, F., Pichler, B.

20th Annual Meeting and Exhibition of the International Society for Magnetic Resonance in Medicine (ISMRM), May 2012 (talk)

ei

Web [BibTex]

Web [BibTex]


no image
Evaluation of a new, large field of view, small animal PET/MR system

Hossain, M., Wehrl, H., Lankes, K., Liu, C., Bezrukov, I., Reischl, G., Pichler, B.

50. Jahrestagung der Deutschen Gesellschaft fuer Nuklearmedizin (NuklearMedizin), April 2012 (talk)

ei

Web [BibTex]

Web [BibTex]


no image
Support Measure Machines for Quasar Target Selection

Muandet, K.

Astro Imaging Workshop, 2012 (talk)

Abstract
In this talk I will discuss the problem of quasar target selection. The objects attributes in astronomy such as fluxes are often subjected to substantial and heterogeneous measurement uncertainties, especially for the medium-redshift between 2.2 and 3.5 quasars which is relatively rare and must be targeted down to g ~ 22 mag. Most of the previous works for quasar target selection includes UV-excess, kernel density estimation, a likelihood approach, and artificial neural network cannot directly deal with the heterogeneous input uncertainties. Recently, extreme deconvolution (XD) has been used to tackle this problem in a well-posed manner. In this work, we present a discriminative approach for quasar target selection that can deal with input uncertainties directly. To do so, we represent each object as a Gaussian distribution whose mean is the object's attribute vector and covariance is the given flux measurement uncertainty. Given a training set of Gaussian distributions, the support measure machines (SMMs) algorithm are trained and used to build the quasar targeting catalog. Preliminary results will also be presented. Joint work with Jo Bovy and Bernhard Sch{\"o}lkopf

ei

Web [BibTex]


no image
PAC-Bayesian Analysis: A Link Between Inference and Statistical Physics

Seldin, Y.

Workshop on Statistical Physics of Inference and Control Theory, 2012 (talk)

ei

Web [BibTex]

Web [BibTex]


no image
PET Performance Measurements of a Next Generation Dedicated Small Animal PET/MR Scanner

Liu, C., Hossain, M., Lankes, K., Bezrukov, I., Wehrl, H., Kolb, A., Judenhofer, M., Pichler, B.

Nuclear Science Symposium and Medical Imaging Conference (NSS-MIC), 2012 (talk)

ei

[BibTex]

[BibTex]


no image
Simultaneous small animal PET/MR reveals different brain networks during stimulation and rest

Wehrl, H., Hossain, M., Lankes, K., Liu, C., Bezrukov, I., Martirosian, P., Reischl, G., Schick, F., Pichler, B.

World Molecular Imaging Congress (WMIC), 2012 (talk)

ei

[BibTex]

[BibTex]


no image
PAC-Bayesian Analysis of Supervised, Unsupervised, and Reinforcement Learning

Seldin, Y., Laviolette, F., Shawe-Taylor, J.

Tutorial at the 29th International Conference on Machine Learning (ICML), 2012 (talk)

ei

Web Web [BibTex]

Web Web [BibTex]


no image
Influence of MR-based attenuation correction on lesions within bone and susceptibility artifact regions

Bezrukov, I., Schmidt, H., Mantlik, F., Schwenzer, N., Brendle, C., Pichler, B.

Molekulare Bildgebung (MoBi), 2012 (talk)

ei

[BibTex]

[BibTex]


no image
Structured Apprenticeship Learning

Boularias, A., Kroemer, O., Peters, J.

European Workshop on Reinforcement Learning (EWRL), 2012 (talk)

ei

[BibTex]

[BibTex]


no image
PAC-Bayesian Analysis and Its Applications

Seldin, Y., Laviolette, F., Shawe-Taylor, J.

Tutorial at The European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD), 2012 (talk)

ei

Web [BibTex]

Web [BibTex]


no image
Machine Learning and Interpretation in Neuroimaging - Revised Selected and Invited Contributions

Langs, G., Rish, I., Grosse-Wentrup, M., Murphy, B.

pages: 266, Springer, Heidelberg, Germany, International Workshop, MLINI, Held at NIPS, 2012, Lecture Notes in Computer Science, Vol. 7263 (proceedings)

ei

DOI [BibTex]

DOI [BibTex]


no image
Kernel Bellman Equations in POMDPs

Nishiyama, Y., Boularias, A., Gretton, A., Fukumizu, K.

Technical Committee on Infomation-Based Induction Sciences and Machine Learning (IBISML'12), 2012 (talk)

ei

[BibTex]

[BibTex]


no image
MICCAI, Workshop on Computational Diffusion MRI, 2012 (electronic publication)

Panagiotaki, E., O’Donnell, L., Schultz, T., Zhang, G.

15th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), Workshop on Computational Diffusion MRI , 2012 (proceedings)

ei

PDF [BibTex]

PDF [BibTex]


no image
Beta oscillations propagate as traveling waves in the macaque prefrontal cortex

Panagiotaropoulos, T., Besserve, M., Logothetis, N.

42nd Annual Meeting of the Society for Neuroscience (Neuroscience), 2012 (talk)

ei

[BibTex]

[BibTex]