Benchmark:

MOT17 | MOT20 |

Short name:

TRCKFRMER_PR

Detector:

Private

Description:

The challenging task of multi-object tracking (MOT) requires simultaneous reasoning about track initialization, identity, and spatiotemporal trajectories. We formulate this task as a frame-to-frame set prediction problem and introduce TrackFormer, an end-to-end MOT approach based on an encoder-decoder Transformer architecture. Our model achieves data association between frames via attention by evolving a set of track predictions through a video sequence. The Transformer decoder initializes new tracks from static object queries and autoregressively follows existing tracks in space and time with the new concept of identity preserving track queries. Both decoder query types benefit from self- and encoder-decoder attention on global frame-level features, thereby omitting any additional graph optimization and matching or modeling of motion and appearance. TrackFormer represents a new tracking-by-attention paradigm and yields state-of-the-art performance on the task of multi-object tracking (MOT17) and segmentation (MOTS20).

Reference:

T. Meinhardt, A. Kirillov, L. Leal-Taixe, C. Feichtenhofer. TrackFormer: Multi-Object Tracking with Transformers. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.

Last submitted:

April 29, 2022 (1 year ago)

Published:

April 29, 2022 at 11:04:08 CET

Submissions:

1

Project page / code:

https://github.com/timmeinhardt/trackformer

Open source:

Yes

Hardware:

7 x 32 GB GPUs

Runtime:

5.7 Hz

Benchmark performance:

Detailed performance:

Raw data:

download