TRCKFRMER_PR: "TrackFormer: Multi-Object Tracking with Transformers" with private detections
Video not available.
Rendering of new sequences is currently deactivated due to heavy load.
Rendering of new sequences is currently deactivated due to heavy load.
Rendering of new sequences is currently deactivated due to heavy load.
Rendering of new sequences is currently deactivated due to heavy load.
Rendering of new sequences is currently deactivated due to heavy load.
Rendering of new sequences is currently deactivated due to heavy load.
Rendering of new sequences is currently deactivated due to heavy load.
Rendering of new sequences is currently deactivated due to heavy load.
Rendering of new sequences is currently deactivated due to heavy load.
Rendering of new sequences is currently deactivated due to heavy load.
Rendering of new sequences is currently deactivated due to heavy load.
Rendering of new sequences is currently deactivated due to heavy load.
Rendering of new sequences is currently deactivated due to heavy load.
Rendering of new sequences is currently deactivated due to heavy load.
Rendering of new sequences is currently deactivated due to heavy load.
Rendering of new sequences is currently deactivated due to heavy load.
Rendering of new sequences is currently deactivated due to heavy load.
Rendering of new sequences is currently deactivated due to heavy load.
Rendering of new sequences is currently deactivated due to heavy load.
Rendering of new sequences is currently deactivated due to heavy load.
Rendering of new sequences is currently deactivated due to heavy load.
Short name:
TRCKFRMER_PR
Detector:
Private
Description:
The challenging task of multi-object tracking (MOT) requires simultaneous reasoning about track initialization, identity, and spatiotemporal trajectories. We formulate this task as a frame-to-frame set prediction problem and introduce TrackFormer, an end-to-end MOT approach based on an encoder-decoder Transformer architecture. Our model achieves data association between frames via attention by evolving a set of track predictions through a video sequence. The Transformer decoder initializes new tracks from static object queries and autoregressively follows existing tracks in space and time with the new concept of identity preserving track queries. Both decoder query types benefit from self- and encoder-decoder attention on global frame-level features, thereby omitting any additional graph optimization and matching or modeling of motion and appearance. TrackFormer represents a new tracking-by-attention paradigm and yields state-of-the-art performance on the task of multi-object tracking (MOT17) and segmentation (MOTS20).
Reference:
T. Meinhardt, A. Kirillov, L. Leal-Taixe, C. Feichtenhofer. TrackFormer: Multi-Object Tracking with Transformers. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Last submitted:
April 29, 2022 (2 years ago)
Published:
April 29, 2022 at 11:04:08 CET
Submissions:
1
Project page / code:
Open source:
Yes
Hardware:
7 x 32 GB GPUs
Runtime:
5.7 Hz
Benchmark performance:
| Sequence | MOTA | IDF1 | HOTA | MT | ML | FP | FN | Rcll | Prcn | AssA | DetA | AssRe | AssPr | DetRe | DetPr | LocA | FAF | ID Sw. | Frag |
| MOT17 | 74.1 | 68.0 | 57.3 | 1,113 (47.3) | 246 (10.4) | 34,602 | 108,777 | 80.7 | 92.9 | 54.1 | 60.9 | 58.0 | 78.8 | 66.8 | 76.9 | 82.8 | 1.9 | 2,829 (0.0) | 4,221 (0.0) |
Detailed performance:
| Sequence | MOTA | IDF1 | HOTA | MT | ML | FP | FN | Rcll | Prcn | AssA | DetA | AssRe | AssPr | DetRe | DetPr | LocA | FAF | ID Sw. | Frag |
| MOT17-01-DPM | 57.9 | 49.7 | 44.2 | 11 | 4 | 477 | 2,191 | 66.0 | 89.9 | 40.9 | 48.2 | 42.9 | 81.1 | 53.5 | 72.9 | 82.1 | 1.1 | 45 | 70 |
| MOT17-01-FRCNN | 57.9 | 49.7 | 44.2 | 11 | 4 | 477 | 2,191 | 66.0 | 89.9 | 40.9 | 48.2 | 42.9 | 81.1 | 53.5 | 72.9 | 82.1 | 1.1 | 45 | 70 |
| MOT17-01-SDP | 57.9 | 49.7 | 44.2 | 11 | 4 | 477 | 2,191 | 66.0 | 89.9 | 40.9 | 48.2 | 42.9 | 81.1 | 53.5 | 72.9 | 82.1 | 1.1 | 45 | 70 |
| MOT17-03-DPM | 88.6 | 79.6 | 66.3 | 124 | 3 | 2,469 | 9,365 | 91.1 | 97.5 | 61.9 | 71.3 | 65.6 | 81.3 | 75.7 | 81.0 | 83.3 | 1.6 | 122 | 316 |
| MOT17-03-FRCNN | 88.6 | 79.6 | 66.3 | 124 | 3 | 2,469 | 9,365 | 91.1 | 97.5 | 61.9 | 71.3 | 65.6 | 81.3 | 75.7 | 81.0 | 83.3 | 1.6 | 122 | 316 |
| MOT17-03-SDP | 88.6 | 79.6 | 66.3 | 124 | 3 | 2,469 | 9,365 | 91.1 | 97.5 | 61.9 | 71.3 | 65.6 | 81.3 | 75.7 | 81.0 | 83.3 | 1.6 | 122 | 316 |
| MOT17-06-DPM | 59.8 | 60.8 | 51.0 | 104 | 27 | 1,791 | 2,775 | 76.5 | 83.4 | 48.1 | 54.4 | 55.0 | 76.0 | 64.4 | 70.3 | 83.8 | 1.5 | 173 | 194 |
| MOT17-06-FRCNN | 59.8 | 60.8 | 51.0 | 104 | 27 | 1,791 | 2,775 | 76.5 | 83.4 | 48.1 | 54.4 | 55.0 | 76.0 | 64.4 | 70.3 | 83.8 | 1.5 | 173 | 194 |
| MOT17-06-SDP | 59.8 | 60.8 | 51.0 | 104 | 27 | 1,791 | 2,775 | 76.5 | 83.4 | 48.1 | 54.4 | 55.0 | 76.0 | 64.4 | 70.3 | 83.8 | 1.5 | 173 | 194 |
| MOT17-07-DPM | 65.6 | 49.5 | 42.8 | 24 | 5 | 1,030 | 4,671 | 72.3 | 92.2 | 34.2 | 54.0 | 38.0 | 71.4 | 59.1 | 75.3 | 81.9 | 2.1 | 118 | 172 |
| MOT17-07-FRCNN | 65.6 | 49.5 | 42.8 | 24 | 5 | 1,030 | 4,671 | 72.3 | 92.2 | 34.2 | 54.0 | 38.0 | 71.4 | 59.1 | 75.3 | 81.9 | 2.1 | 118 | 172 |
| MOT17-07-SDP | 65.6 | 49.5 | 42.8 | 24 | 5 | 1,030 | 4,671 | 72.3 | 92.2 | 34.2 | 54.0 | 38.0 | 71.4 | 59.1 | 75.3 | 81.9 | 2.1 | 118 | 172 |
| MOT17-08-DPM | 54.5 | 42.5 | 39.2 | 24 | 9 | 1,461 | 7,861 | 62.8 | 90.1 | 33.0 | 47.5 | 36.1 | 71.1 | 51.8 | 74.4 | 81.5 | 2.3 | 279 | 320 |
| MOT17-08-FRCNN | 54.5 | 42.5 | 39.2 | 24 | 9 | 1,461 | 7,861 | 62.8 | 90.1 | 33.0 | 47.5 | 36.1 | 71.1 | 51.8 | 74.4 | 81.5 | 2.3 | 279 | 320 |
| MOT17-08-SDP | 54.5 | 42.5 | 39.2 | 24 | 9 | 1,461 | 7,861 | 62.8 | 90.1 | 33.0 | 47.5 | 36.1 | 71.1 | 51.8 | 74.4 | 81.5 | 2.3 | 279 | 320 |
| MOT17-12-DPM | 51.8 | 63.0 | 53.5 | 43 | 14 | 1,880 | 2,258 | 73.9 | 77.3 | 57.1 | 50.4 | 62.6 | 79.9 | 63.0 | 65.8 | 84.6 | 2.1 | 42 | 87 |
| MOT17-12-FRCNN | 51.8 | 63.0 | 53.5 | 43 | 14 | 1,880 | 2,258 | 73.9 | 77.3 | 57.1 | 50.4 | 62.6 | 79.9 | 63.0 | 65.8 | 84.6 | 2.1 | 42 | 87 |
| MOT17-12-SDP | 51.8 | 63.0 | 53.5 | 43 | 14 | 1,880 | 2,258 | 73.9 | 77.3 | 57.1 | 50.4 | 62.6 | 79.9 | 63.0 | 65.8 | 84.6 | 2.1 | 42 | 87 |
| MOT17-14-DPM | 47.4 | 54.9 | 41.3 | 41 | 20 | 2,426 | 7,138 | 61.4 | 82.4 | 41.1 | 42.2 | 44.3 | 73.8 | 48.8 | 65.5 | 79.7 | 3.2 | 164 | 248 |
| MOT17-14-FRCNN | 47.4 | 54.9 | 41.3 | 41 | 20 | 2,426 | 7,138 | 61.4 | 82.4 | 41.1 | 42.2 | 44.3 | 73.8 | 48.8 | 65.5 | 79.7 | 3.2 | 164 | 248 |
| MOT17-14-SDP | 47.4 | 54.9 | 41.3 | 41 | 20 | 2,426 | 7,138 | 61.4 | 82.4 | 41.1 | 42.2 | 44.3 | 73.8 | 48.8 | 65.5 | 79.7 | 3.2 | 164 | 248 |
Raw data:
