Siyu Tang

Research Group Leader

Perceiving Systems

Office: N3.009
Max-Planck-Ring 4
72076 Tübingen
Germany

I will be joining the Computer Science Department of ETH Zürich as a tenure track assistant professor from January 2020.

I am leading the Holistic Vision Group (HVG) in the Department of Perceiving Systems at the Max Planck Institute for Intelligent Systems, my group is funded by the DFG through the CRC 1233 on Robust Vision.

I am interested in the intersection between computer vision and machine learning with a focus on holistic visual scene understanding. In particular, I am interested in analyzing and modeling people in our complex visual scenes.

Offers: I am looking for highly motivated PhD interns. I also have projects for bachelor and master thesis. If you are interested, please contact me direclty or send your application to ps-apply@tuebingen.mpg.de

I was a postdoctoral researcher at the Max Planck Institute for Intelligent Systems, advised by Michael Black. I finished my PhD summa cum laude at the Max Planck Institute for Informatics, under the supervision of Prof. Bernt Schiele. Before that, I received Master degree in Computer Science at RWTH Aachen University, advised by Prof. Bastian Leibe and Bachelor degree in Computer Science and Technology Department at Zhejiang University, China. I was a research intern at National Institute of Informatics, under the supervision of Prof. Helmut Prendinger.

News:

New! I will be an area chair for CVPR 2020 and tutorial chair for CVPR 2023 at Vancouver.
New! Our work on Low-rank Random Tensor for Bilinear Pooling is online.
Our EU grant proposal CLIPE is accepted for funding, call for PhD students will come soon.
I'm co-organizing the 3rd Computational Vision Summer School this year. We have amazing speakers. Apply here.
Our work on learning bilinear pooling for fine-grained action parsing is accepted to CVPR 2019!
Two internship positions on Kernel Methods Meet Deep Neural Networks to be filled until 03/2019. Apply here!
I received the DAGM MVTec 2018 Dissertation Award.
We have one paper accepted to TPAMI.
We have one paper accepted to ACCV 2018 as oral presentation.
One paper accepted to ECCV 2018.
One paper accepted to BMVC 2018.
Our work on part-aligned bilinear representations for person re-identification is online.
Our work on human action segmentation in real time is online, and the code is available.
I will be an area chair for ACCV 2018.
I received an Early career research grant to start my own research group at the Max Planck Instiute for Intelligent Systems and the University of Tübingen, details coming soon. I am looking for highly motivated PhD student and PhD interns!
I have successfully defended my PhD thesis "People Detection and Tracking in Crowded Scenes" on the 29th September 2017 at the Max Planck Institute for Informatics. Thesis Committee: Prof. Bernt Schiele, Prof. Michael Black, Prof. Luc Van Gool.
Winner of the CVPR 2017 Multi-Object Tracking Challenge (MOT17).
Four papers accepted at CVPR 2017!

Awards:

DAGM MVTec Dissertation Award, 2018
Winner of the Multi-Object Tracking Challenge at CVPR 2017
Winner of the Multi-Object Tracking Challenge at ECCV 2016
BMVC Best Paper Award, 2012
Scholarship for excellence in academic performance RWTH Aachen 2009, 2010

Teaching:

SS 2016: High-Level Computer Vision, Saarland University, teaching assistant
SS 2015: High-Level Computer Vision, Saarland University, teaching assistant
SS 2013: High-Level Computer Vision, Saarland University, teaching assistant

Reviewing:

IJCV, TPAMI

CVPR 2016, ECCV 2016,

CVPR 2017, ICCV 2017, NIPS 2017, ICML 2017.

PhD Student

Qianli Ma (with Michael Black)

Visitor

Yan Zhang

Oct 2018 - March 2019

Master Student

David Hoffmann (with Dimitris Tzionas)

April 2018 - present

Past Visitor

Jie Song

June 2017 - December 2017

Efficient and Scalable Inference

Machine Learning is an important tool for Computer Vision. After the success of Deep Neural Networks(DNN)s in image classification tasks, many other tasks were solved using DNNs. However, working with deep learning, researchers are confronted with many problems related to implementation, hyper-parameter se...

Siyu Tang Christoph Lassner Daniel Kappler Martin Kiefel Peter Vincent Gehler Maren Mahsereci Lukas Balles Philipp Hennig Javier Romero Philipp Hennig

Learning from Synthetic Data

Deep learning has brought rapid progress for many computer vision problems but current methods require large training datasets with annotated ground truth. Human annotators tend to be reasonably efficient for tasks like sparse 2D joint estimation, however annotation for other tasks like dense optical...

Javier Romero Anurag Ranjan Michael Black Jonas Wulff David Hoffmann Dimitrios Tzionas Siyu Tang Naureen Mahmood

Action and Behavior

Human behavior can be described at multiple levels. At the lowest level, we observe the 3D pose of the body over time. Poses can be organized into primitives that capture coordinated activity of different body parts. These further form more complex "actions" or "behaviors". Finally, und...

Siyu Tang Yan Zhang

Groups and Crowds

People are often a central element of visual scenes. It has been a long-standing goal in computer vision to develop computational models that enable machines to detect crowds of people, analyze their motion and poses, infer their actions and reason about the consequences. Our research addresses a wide rang...

Siyu Tang Michael Black Peter Vincent Gehler

21 results (View BibTeX file of all listed publications)

2019

Local Temporal Bilinear Pooling for Fine-grained Action Parsing

Zhang, Y., Tang, S., Muandet, K., Jarvers, C., Neumann, H.

In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), IEEE International Conference on Computer Vision and Pattern Recognition (CVPR) 2019, June 2019 (inproceedings)

Abstract

Fine-grained temporal action parsing is important in many applications, such as daily activity understanding, human motion analysis, surgical robotics and others requiring subtle and precise operations in a long-term period. In this paper we propose a novel bilinear pooling operation, which is used in intermediate layers of a temporal convolutional encoder-decoder net. In contrast to other work, our proposed bilinear pooling is learnable and hence can capture more complex local statistics than the conventional counterpart. In addition, we introduce exact lower-dimension representations of our bilinear forms, so that the dimensionality is reduced with neither information loss nor extra computation. We perform intensive experiments to quantitatively analyze our model and show the superior performances to other state-of-the-art work on various datasets.

ei ps

Code video demo pdf link (url) [BibTex]

2019

ei ps Zhang, Y., Tang, S., Muandet, K., Jarvers, C., Neumann, H. Local Temporal Bilinear Pooling for Fine-grained Action Parsing In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), IEEE International Conference on Computer Vision and Pattern Recognition (CVPR) 2019, June 2019 (inproceedings)

Code video demo pdf link (url) [BibTex]

Learning to Train with Synthetic Humans

Hoffmann, D., Tzionas, D., Black, M., Tang, S.

In 2019 (inproceedings)

Abstract

Neural networks need big annotated datasets for training. However, manual annotation can be too expensive or even unfeasible for certain tasks, like multi-person 2D pose estimation with severe occlusions. A remedy for this is synthetic data with perfect ground truth. Here we explore two variations of synthetic data for this challenging problem; a dataset with purely synthetic humans, as well as a real dataset augmented with synthetic humans. We then study which approach better generalizes to real data, as well as the influence of virtual humans in the training loss. We observe that not all synthetic samples are equally informative for training, while the informative samples are different for each training stage. To exploit this observation, we employ an adversarial student-teacher framework; the teacher improves the student by providing the hardest samples for its current state as a challenge. Experiments show that this student-teacher framework outperforms all our baselines.

[BibTex]

ps Hoffmann, D., Tzionas, D., Black, M., Tang, S. Learning to Train with Synthetic Humans In 2019 (inproceedings)

[BibTex]

2018

Customized Multi-Person Tracker

Ma, L., Tang, S., Black, M. J., Gool, L. V.

In Computer Vision – ACCV 2018, Springer International Publishing, Asian Conference on Computer Vision, December 2018 (inproceedings)

PDF Project Page [BibTex]

2018

ps Ma, L., Tang, S., Black, M. J., Gool, L. V. Customized Multi-Person Tracker In Computer Vision – ACCV 2018, Springer International Publishing, Asian Conference on Computer Vision, December 2018 (inproceedings)

PDF Project Page [BibTex]

Human Motion Parsing by Hierarchical Dynamic Clustering

Zhang, Y., Tang, S., Sun, H., Neumann, H.

In Proceedings of the British Machine Vision Conference (BMVC), pages: 269, BMVA Press, 29th British Machine Vision Conference, September 2018 (inproceedings)

Abstract

Parsing continuous human motion into meaningful segments plays an essential role in various applications. In this work, we propose a hierarchical dynamic clustering framework to derive action clusters from a sequence of local features in an unsuper- vised bottom-up manner. We systematically investigate the modules in this framework and particularly propose diverse temporal pooling schemes, in order to realize accurate temporal action localization. We demonstrate our method on two motion parsing tasks: temporal action segmentation and abnormal behavior detection. The experimental results indicate that the proposed framework is significantly more effective than the other related state-of-the-art methods on several datasets.

pdf Project Page [BibTex]

ps Zhang, Y., Tang, S., Sun, H., Neumann, H. Human Motion Parsing by Hierarchical Dynamic Clustering In Proceedings of the British Machine Vision Conference (BMVC), pages: 269, BMVA Press, 29th British Machine Vision Conference, September 2018 (inproceedings)

pdf Project Page [BibTex]

Part-Aligned Bilinear Representations for Person Re-identification

Suh, Y., Wang, J., Tang, S., Mei, T., Lee, K. M.

In European Conference on Computer Vision (ECCV), 11218, pages: 418-437, Springer, Cham, September 2018 (inproceedings)

Abstract

Comparing the appearance of corresponding body parts is essential for person re-identification. However, body parts are frequently misaligned be- tween detected boxes, due to the detection errors and the pose/viewpoint changes. In this paper, we propose a network that learns a part-aligned representation for person re-identification. Our model consists of a two-stream network, which gen- erates appearance and body part feature maps respectively, and a bilinear-pooling layer that fuses two feature maps to an image descriptor. We show that it results in a compact descriptor, where the inner product between two image descriptors is equivalent to an aggregation of the local appearance similarities of the cor- responding body parts, and thereby significantly reduces the part misalignment problem. Our approach is advantageous over other pose-guided representations by learning part descriptors optimal for person re-identification. Training the net- work does not require any part annotation on the person re-identification dataset. Instead, we simply initialize the part sub-stream using a pre-trained sub-network of an existing pose estimation network and train the whole network to minimize the re-identification loss. We validate the effectiveness of our approach by demon- strating its superiority over the state-of-the-art methods on the standard bench- mark datasets including Market-1501, CUHK03, CUHK01 and DukeMTMC, and standard video dataset MARS.

pdf supplementary DOI Project Page [BibTex]

ps Suh, Y., Wang, J., Tang, S., Mei, T., Lee, K. M. Part-Aligned Bilinear Representations for Person Re-identification In European Conference on Computer Vision (ECCV), 11218, pages: 418-437, Springer, Cham, September 2018 (inproceedings)

pdf supplementary DOI Project Page [BibTex]

End-to-end Learning for Graph Decomposition

Song, J., Andres, B., Black, M., Hilliges, O., Tang, S.

arXiv:1812.09737, 2018 (article)

Abstract

We propose a novel end-to-end trainable framework for the graph decomposition problem. The minimum cost mul- ticut problem is first converted to an unconstrained binary cubic formulation where cycle consistency constraints are incorporated into the objective function. The new optimiza- tion problem can be viewed as a Conditional Random Field (CRF) in which the random variables are associated with the binary edge labels of the initial graph and the hard con- straints are introduced in the CRF as high-order potentials. The parameters of a standard Neural Network and the fully differentiable CRF are optimized in an end-to-end manner. Furthermore, our method utilizes the cycle constraints as meta-supervisory signals during the learning of the deep feature representations by taking the dependencies between the output random variables into account. We present analy- ses of the end-to-end learned representations, showing the impact of the joint training, on the task of clustering images of MNIST. We also validate the effectiveness of our approach both for the feature learning and the final clustering on the challenging task of real-world multi-person pose estimation

paper.pdf Video link (url) [BibTex]

ps Song, J., Andres, B., Black, M., Hilliges, O., Tang, S. End-to-end Learning for Graph Decomposition arXiv:1812.09737, 2018 (article)

paper.pdf Video link (url) [BibTex]

Temporal Human Action Segmentation via Dynamic Clustering

Zhang, Y., Sun, H., Tang, S., Neumann, H.

arXiv preprint arXiv:1803.05790, 2018 (article)

Abstract

We present an effective dynamic clustering algorithm for the task of temporal human action segmentation, which has comprehensive applications such as robotics, motion analysis, and patient monitoring. Our proposed algorithm is unsupervised, fast, generic to process various types of features, and applica- ble in both the online and offline settings. We perform extensive experiments of processing data streams, and show that our algorithm achieves the state-of- the-art results for both online and offline settings.

link (url) [BibTex]

ps Zhang, Y., Sun, H., Tang, S., Neumann, H. Temporal Human Action Segmentation via Dynamic Clustering arXiv preprint arXiv:1803.05790, 2018 (article)

link (url) [BibTex]

Motion Segmentation & Multiple Object Tracking by Correlation Co-Clustering

Keuper, M., Tang, S., Andres, B., Brox, T., Schiele, B.

IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018 (article)

pdf DOI Project Page [BibTex]

ps Keuper, M., Tang, S., Andres, B., Brox, T., Schiele, B. Motion Segmentation & Multiple Object Tracking by Correlation Co-Clustering IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018 (article)

pdf DOI Project Page [BibTex]

2017

Joint Graph Decomposition and Node Labeling by Local Search

Levinkov, E., Uhrig, J., Tang, S., Omran, M., Insafutdinov, E., Kirillov, A., Rother, C., Brox, T., Schiele, B., Andres, B.

In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages: 1904-1912, IEEE, IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), July 2017 (inproceedings)

PDF Supplementary DOI Project Page [BibTex]

2017

ps Levinkov, E., Uhrig, J., Tang, S., Omran, M., Insafutdinov, E., Kirillov, A., Rother, C., Brox, T., Schiele, B., Andres, B. Joint Graph Decomposition and Node Labeling by Local Search In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages: 1904-1912, IEEE, IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), July 2017 (inproceedings)

PDF Supplementary DOI Project Page [BibTex]

Articulated Multi-person Tracking in the Wild

Insafutdinov, E., Andriluka, M., Pishchulin, L., Tang, S., Levinkov, E., Andres, B., Schiele, B.

In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages: 1293-1301, IEEE, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017, Oral (inproceedings)

DOI [BibTex]

ps Insafutdinov, E., Andriluka, M., Pishchulin, L., Tang, S., Levinkov, E., Andres, B., Schiele, B. Articulated Multi-person Tracking in the Wild In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages: 1293-1301, IEEE, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017, Oral (inproceedings)

DOI [BibTex]

Multiple People Tracking by Lifted Multicut and Person Re-identification

Tang, S., Andriluka, M., Andres, B., Schiele, B.

In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages: 3701-3710, IEEE Computer Society, Washington, DC, USA, IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), July 2017 (inproceedings)

DOI Project Page [BibTex]

ps Tang, S., Andriluka, M., Andres, B., Schiele, B. Multiple People Tracking by Lifted Multicut and Person Re-identification In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages: 3701-3710, IEEE Computer Society, Washington, DC, USA, IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), July 2017 (inproceedings)

DOI Project Page [BibTex]

Generating Descriptions with Grounded and Co-Referenced People

Rohrbach, A., Rohrbach, M., Tang, S., Oh, S. J., Schiele, B.

In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages: 4196-4206, IEEE, Piscataway, NJ, USA, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017 (inproceedings)

PDF DOI Project Page [BibTex]

ps Rohrbach, A., Rohrbach, M., Tang, S., Oh, S. J., Schiele, B. Generating Descriptions with Grounded and Co-Referenced People In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages: 4196-4206, IEEE, Piscataway, NJ, USA, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017 (inproceedings)

PDF DOI Project Page [BibTex]

2016

DeepCut: Joint Subset Partition and Labeling for Multi Person Pose Estimation

Pishchulin, L., Insafutdinov, E., Tang, S., Andres, B., Andriluka, M., Gehler, P., Schiele, B.

In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages: 4929-4937, IEEE, IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), June 2016 (inproceedings)

Abstract

This paper considers the task of articulated human pose estimation of multiple people in real-world images. We propose an approach that jointly solves the tasks of detection and pose estimation: it infers the number of persons in a scene, identifies occluded body parts, and disambiguates body parts between people in close proximity of each other. This joint formulation is in contrast to previous strategies, that address the problem by first detecting people and subsequently estimating their body pose. We propose a partitioning and labeling formulation of a set of body-part hypotheses generated with CNN-based part detectors. Our formulation, an instance of an integer linear program, implicitly performs non-maximum suppression on the set of part candidates and groups them to form configurations of body parts respecting geometric and appearance constraints. Experiments on four different datasets demonstrate state-of-the-art results for both single person and multi person pose estimation.

code pdf supplementary DOI Project Page [BibTex]

2016

ps Pishchulin, L., Insafutdinov, E., Tang, S., Andres, B., Andriluka, M., Gehler, P., Schiele, B. DeepCut: Joint Subset Partition and Labeling for Multi Person Pose Estimation In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages: 4929-4937, IEEE, IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), June 2016 (inproceedings)

code pdf supplementary DOI Project Page [BibTex]

Multi-Person Tracking by Multicuts and Deep Matching

(Winner of the Multi-Object Tracking Challenge ECCV 2016)

Tang, S., Andres, B., Andriluka, M., Schiele, B.

ECCV Workshop on Benchmarking Mutliple Object Tracking, 2016 (conference)

PDF [BibTex]

ps Tang, S., Andres, B., Andriluka, M., Schiele, B. Multi-Person Tracking by Multicuts and Deep Matching ECCV Workshop on Benchmarking Mutliple Object Tracking, 2016 (conference)

PDF [BibTex]

A Multi-cut Formulation for Joint Segmentation and Tracking of Multiple Objects

Keuper, M., Tang, S., Yu, Z., Andres, B., Brox, T., Schiele, B.

In arXiv:1607.06317, 2016 (inproceedings)

PDF [BibTex]

ps Keuper, M., Tang, S., Yu, Z., Andres, B., Brox, T., Schiele, B. A Multi-cut Formulation for Joint Segmentation and Tracking of Multiple Objects In arXiv:1607.06317, 2016 (inproceedings)

PDF [BibTex]

2015

Subgraph decomposition for multi-target tracking

Tang, S., Andres, B., Andriluka, M., Schiele, B.

In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages: 5033-5041, IEEE, IEEE Conference on Computer Vision and Pattern Recognition, June 2015 (inproceedings)

PDF Proof-of-Lemma-1 DOI [BibTex]

2015

ps Tang, S., Andres, B., Andriluka, M., Schiele, B. Subgraph decomposition for multi-target tracking In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages: 5033-5041, IEEE, IEEE Conference on Computer Vision and Pattern Recognition, June 2015 (inproceedings)

PDF Proof-of-Lemma-1 DOI [BibTex]

2014

Learning People Detectors for Tracking in Crowded Scenes.

Tang, S., Andriluka, M., Milan, A., Schindler, K., Roth, S., Schiele, B.

2014, Scene Understanding Workshop (SUNw, CVPR workshop) (unpublished)

[BibTex]

2014

ps Tang, S., Andriluka, M., Milan, A., Schindler, K., Roth, S., Schiele, B. Learning People Detectors for Tracking in Crowded Scenes. 2014, Scene Understanding Workshop (SUNw, CVPR workshop) (unpublished)

[BibTex]

Detection and Tracking of Occluded People

Tang, S., Andriluka, M., Schiele, B.

International Journal of Computer Vision, 110, pages: 58-69, 2014 (article)

PDF [BibTex]

ps Tang, S., Andriluka, M., Schiele, B. Detection and Tracking of Occluded People International Journal of Computer Vision, 110, pages: 58-69, 2014 (article)

PDF [BibTex]

2013

Learning People Detectors for Tracking in Crowded Scenes

Tang, S., Andriluka, M., Milan, A., Schindler, K., Roth, S., Schiele, B.

In 2013 IEEE International Conference on Computer Vision, pages: 1049-1056, IEEE, IEEE International Conference on Computer Vision, December 2013 (inproceedings)

PDF DOI [BibTex]

2013

ps Tang, S., Andriluka, M., Milan, A., Schindler, K., Roth, S., Schiele, B. Learning People Detectors for Tracking in Crowded Scenes In 2013 IEEE International Conference on Computer Vision, pages: 1049-1056, IEEE, IEEE International Conference on Computer Vision, December 2013 (inproceedings)

PDF DOI [BibTex]

2012

Detection and Tracking of Occluded People

(Best Paper Award)

Tang, S., Andriluka, M., Schiele, B.

In British Machine Vision Conference (BMVC), 2012, BMVC Best Paper Award (inproceedings)

PDF [BibTex]

2012

ps Tang, S., Andriluka, M., Schiele, B. Detection and Tracking of Occluded People In British Machine Vision Conference (BMVC), 2012, BMVC Best Paper Award (inproceedings)

PDF [BibTex]

2011

OpenBioSafetyLab: A virtual world based biosafety training application for medical students

Nakasone, A., Tang, S., Shigematsu, M., Heinecke, B., Fujimoto, S., Prendinger, H.

In International Conference on Information Technology: New Generations (ITNG), IEEE CPS, 2011 (inproceedings)

PDF [BibTex]

2011

ps Nakasone, A., Tang, S., Shigematsu, M., Heinecke, B., Fujimoto, S., Prendinger, H. OpenBioSafetyLab: A virtual world based biosafety training application for medical students In International Conference on Information Technology: New Generations (ITNG), IEEE CPS, 2011 (inproceedings)

PDF [BibTex]

Siyu Tang

News:

Awards:

Teaching:

Reviewing:

PhD Student

Visitor

Master Student

Past Visitor

2019

2019

2018

2018

2017

2017

2016

2016

2015

2015

2014

2014

2013

2013

2012

2012

2011

2011

Latest News

Links

Contact Us