Understanding actions in videos goes beyond recognizing appearances - it requires tracking objects as they perform complex behaviors over time. While recent benchmarks have made strides in temporal action localization and referring object tracking, they often treat these tasks in isolation. A gap remains in evaluating models that must both localize and track multiple objects based on action-driven, natural language queries.

At the 8th edition of the BMTT workshop, we focus on action-aware multi-object tracking, aiming to bridge the divide between vision and language by introducing unified challenges that evaluate both temporal localization and object tracking. We invite the community to examine whether current models can reason about actions, follow fine-grained language instructions, and scale to more complex, real-world scenes.

Through invited talks, the challenge, and open discussions, this workshop will explore the limitations of existing vision-language models, analyze the performance of state-of-the-art trackers and temporal localizers on action-based queries, and promote the development of more robust, multimodal video understanding systems.

User Instructed Spatiotemporal Detections. It shows the bounding boxes of the referred objects over time for a given set of queries. The top shows the current frame number and the given set of queries, and the bottom shows the period when the object appears. The full video is available here.

Info