This page display the submited results for Tracking Leaderboards. For each submission, we display several
main metrics in main table. For detailed information, more metrics, per-sequence results and visualisation
(coming soon),
please click submission name. For all tables, you can click headers to sort the results. Note you can
download the submission zip file as well. Legends, metrics
descriptions and reference are displayed after leaderboards table.
The problem of tracking multiple objects in a video se- quence poses several challenging tasks. For tracking-by- detection, these include object re-identification, motion pre- diction and dealing with occlusions. We present a tracker (without bells and whistles) that accomplishes tracking without specifically targeting any of these tasks, in partic- ular, we perform no training or optimization on tracking data. To this end, we exploit the bounding box regression of an object detector to predict the position of an object in the next frame, thereby converting a detector into a Tracktor. We demonstrate the potential of Tracktor and provide a new state-of-the-art on three multi-object tracking benchmarks by extending it with a straightforward re-identification and camera motion compensation. We then perform an analysis on the performance and failure cases of several state-of-the-art tracking methods in comparison to our Tracktor. Surprisingly, none of the dedicated tracking methods are considerably better in deal- ing with complex tracking scenarios, namely, small and occluded objects or missing detections. However, our ap- proach tackles most of the easy tracking scenarios. There- fore, we motivate our approach as a new tracking paradigm and point out promising future research directions. Over- all, Tracktor yields superior tracking performance than any current tracking method and our analysis exposes remain- ing and unsolved tracking challenges to inspire future re- search directions.
Panoramic image detection and tracking using both image and point clouds. For tracking, we adopt the Cascade rcnn as our object detector. The backbone of the detector is ResNet 101. The detector is pre-trained on the COCO dataset and then fine tuned on the JRDB dataset. During training, we employ data augment methods such as mixup, random crop, multiscale training to augment the dataset. During inference, we employ horizontal flip and softnms techniques to generate more robust resutls. For tracking, we fuse both the 3D image and 3D point clouds for robust tracking.
Publication title
Know Your Surroundings: Panoramic Multi-Object Tracking by Multimodality Collaboration
The problem of tracking multiple objects in a video se- quence poses several challenging tasks. For tracking-by- detection, these include object re-identification, motion pre- diction and dealing with occlusions. We present a tracker (without bells and whistles) that accomplishes tracking without specifically targeting any of these tasks, in partic- ular, we perform no training or optimization on tracking data. To this end, we exploit the bounding box regression of an object detector to predict the position of an object in the next frame, thereby converting a detector into a Tracktor. We demonstrate the potential of Tracktor and provide a new state-of-the-art on three multi-object tracking benchmarks by extending it with a straightforward re-identification and camera motion compensation. We then perform an analysis on the performance and failure cases of several state-of-the-art tracking methods in comparison to our Tracktor. Surprisingly, none of the dedicated tracking methods are considerably better in deal- ing with complex tracking scenarios, namely, small and occluded objects or missing detections. However, our ap- proach tackles most of the easy tracking scenarios. There- fore, we motivate our approach as a new tracking paradigm and point out promising future research directions. Over- all, Tracktor yields superior tracking performance than any current tracking method and our analysis exposes remain- ing and unsolved tracking challenges to inspire future re- search directions.
Simple Online and Realtime Tracking (SORT) is a pragmatic approach to multiple object tracking with a focus on simple, effective algorithms. In this paper, we integrate appearance information to improve the performance of SORT. Due to this extension we are able to track objects through longer periods of occlusions, effectively reducing the number of identity switches. In spirit of the original framework we place much of the computational complexity into an offline pre-training stage where we learn a deep association metric on a largescale person re-identification dataset. During online application, we establish measurement-to-track associations using nearest neighbor queries in visual appearance space. Experimental evaluation shows that our extensions reduce the number of identity switches by 45%, achieving overall competitive performance at high frame rates.
Simple Online and Realtime Tracking (SORT) is a pragmatic approach to multiple object tracking with a focus on simple, effective algorithms. In this paper, we integrate appearance information to improve the performance of SORT. Due to this extension we are able to track objects through longer periods of occlusions, effectively reducing the number of identity switches. In spirit of the original framework we place much of the computational complexity into an offline pre-training stage where we learn a deep association metric on a large-scale person re-identification dataset. During online application, we establish measurement-to-track associations using nearest neighbor queries in visual appearance space. Experimental evaluation shows that our extensions reduce the number of identity switches by 45%, achieving overall competitive performance at high frame rates.
Publication title
Simple Online and Realtime Tracking with a Deep Association Metric
A robotic agent navigating autonomously needs to perceive and track the motion of objects and other agents in its surroundings to be able to plan and execute robust and safe trajectories. To be useful for planning and execution, the motion should be perceived in 3D Cartesian space. However, most recent multi-object tracking (MOT) research have focused on tracking people and moving objects in 2D RGB video sequences. This is because searching in 3D is costly, there is a lack of annotated datasets with 3D labels of moving agents and a relative scarcity of data with 3D sensor modalities. In this work we present JRMOT, a novel 3D MOT system that integrates information from RGB images and 3D point clouds to achieve real-time, state-of-the-art tracking performance. Our system incorporates recent advancements in neural network based re-identification as well as 2D and 3D detectors and descriptors. We integrate them into a joint probabilistic data-association framework within a multi-modal recursive Kalman architecture to achieve online, real-time 3D MOT. As part of our work, we release the JRDB dataset, a novel large scale 2D+3D dataset and benchmark annotated with over 2 million boxes and 3500 time consistent 2D+3D trajectories across 54 indoor and outdoor scenes. This dataset, that we use to train and test our model, contains over 60 minutes of data including 360 degree cylindrical RGB video and 3D pointclouds, in social settings. The presented 3D MOT system demonstrates state-ofthe-art performance against competing methods on the popular 2D tracking KITTI benchmark and serves as a competitive 3D tracking baseline for our dataset and benchmark. Moreover, our tests on our social robot JackRabbot indicate that the system is capable of tracking multiple pedestrians fast and reliably.
Publication title
JRMOT: A Real-Time 3D Multi-Object Tracker and a New Large-Scale Dataset
Publication authors:
A. Shenoi, M. Patel, J. Gwak, P. Goebel, A. Sadeghian, H. Rezatofighi, R. Martín-Martín and S. Savar
Tracking-by-detection is a common approach to multiobject tracking. With ever increasing performances of object detectors, the basis for a tracker becomes much more reliable. In combination with commonly higher frame rates, this poses a shift in the challenges for a successful tracker. That shift enables the deployment of much simpler tracking algorithms which can compete with more sophisticated approaches at a fraction of the computational cost. We present such an algorithm and show with thorough experiments its potential using a wide range of object detectors. The proposed method can easily run at 100K fps while outperforming the state-of-the-art on the DETRAC vehicle tracking dataset
The problem of tracking multiple objects in a video sequence poses several challenging tasks. For tracking-bydetection, these include object re-identification, motion prediction and dealing with occlusions. We present a tracker (without bells and whistles) that accomplishes tracking without specifically targeting any of these tasks, in particular, we perform no training or optimization on tracking data. To this end, we exploit the bounding box regression of an object detector to predict the position of an object in the next frame, thereby converting a detector into a Tracktor. We demonstrate the potential of Tracktor and provide a new state-of-the-art on three multi-object tracking benchmarks by extending it with a straightforward re-identification and camera motion compensation. We then perform an analysis on the performance and failure cases of several state-of-the-art tracking methods in comparison to our Tracktor. Surprisingly, none of the dedicated tracking methods are considerably better in dealing with complex tracking scenarios, namely, small and occluded objects or missing detections. However, our approach tackles most of the easy tracking scenarios. Therefore, we motivate our approach as a new tracking paradigm and point out promising future research directions. Overall, Tracktor yields superior tracking performance than any current tracking method and our analysis exposes remaining and unsolved tracking challenges to inspire future research directions.
With the increasing prevalence of robots in daily life, it is crucial to enable robots to construct a reliable map online to navigate in unbounded and changing environments. Although existing methods can individually achieve the goals of spatial mapping and dynamic object detection and tracking, limited research has been conducted on an effective combination of these two important abilities. The proposed framework, SMAT (Simultaneous Mapping and Tracking), integrates the front-end dynamic object detection and tracking module with the back-end static mapping module using a self-reinforcing mechanism, which promotes mutual improvement of mapping and tracking performance. The conducted experiments demonstrate the framework's effectiveness in real-world applications, achieving successful long-range navigation and mapping in multiple urban environments using only one LiDAR, a CPU-only onboard computer, and a consumer-level GPS receiver.
Publication title
N/A
Publication authors:
N/A
Publication venue and year
N/A
Publication URL:
N/A
Code Language
C++
Hardware
i5 8259u
Code Website
https://sites.google.com/view/smat-nav
Code URL
N/A
Visualisation
Best
3 By
MOTA ↓
Generating
hewlett-class-2019-01-23_1, Come Back Later!
Multiple object tracking in 3D point clouds hasapplications in mobile robots and autonomous driving. This is achallenging problem due to the sparse nature of the point cloudsand the added difficulty of annotation in 3D for supervisedlearning. To overcome these challenges, we propose a neuralnetwork architecture that learns effective object features andtheir affinities in a self-supervised fashion for multiple objecttracking in 3D point clouds captured with LiDAR sensors. Forself supervision, we use two approaches. First, we generatetwo augmented LiDAR frames from a single real frame byapplying translation, rotation and cutout to the objects. Second,we synthesize a LiDAR frame using CAD models or primitivegeometric shapes and then apply the above three augmenta-tions to them. Hence, the ground truth object locations andassociations are known in both frames for self supervision.This removes the need to annotate object associations in realdata, and additionally the need for training data collection andannotation for object detection in synthetic data. To the best ofour knowledge, this is the first self supervised multiple objecttracking in 3D method. Our model achieves state of the artresults.
Publication title
Self Supervised Learning for Multiple Object Tracking in 3D Point Clouds
Publication authors:
Aakash Kumar, Jyoti Kini, Ajmal Mian, Mubarak Shah
Publication venue and year
2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) October 23-27, 2022,
In recent times, the scope of LIDAR (Light Detection and Ranging) sensor-based technology has spread across numerous fields. It is popularly used to map terrain and navigation information into reliable 3D point cloud data, potentially revolutionizing the autonomous vehicles and assistive robotic industry. A point cloud is a dense compilation of spatial data in 3D coordinates. It plays a vital role in modeling complex real-world scenes since it preserves structural information and avoids perspective distortion, unlike image data, which is the projection of a 3D structure on a 2D plane. In order to leverage the intrinsic capabilities of the LIDAR data, we propose a PointNet-based approach for 3D Multi-Object Tracking (MOT).
Publication title
PC-DAN: Point Cloud based Deep Affinity Network for 3D Multi-Object Tracking
A robotic agent navigating autonomously needs to perceive and track the motion of objects and other agents in its surroundings to be able to plan and execute robust and safe trajectories. To be useful for planning and execution, the motion should be perceived in 3D Cartesian space. However, most recent multi-object tracking (MOT) research have focused on tracking people and moving objects in 2D RGB video sequences. This is because searching in 3D is costly, there is a lack of annotated datasets with 3D labels of moving agents and a relative scarcity of data with 3D sensor modalities. In this work we present JRMOT, a novel 3D MOT system that integrates information from RGB images and 3D point clouds to achieve real-time, state-of-the-art tracking performance. Our system incorporates recent advancements in neural network based re-identification as well as 2D and 3D detectors and descriptors. We integrate them into a joint probabilistic data-association framework within a multi-modal recursive Kalman architecture to achieve online, real-time 3D MOT. As part of our work, we release the JRDB dataset, a novel large scale 2D+3D dataset and benchmark annotated with over 2 million boxes and 3500 time consistent 2D+3D trajectories across 54 indoor and outdoor scenes. This dataset, that we use to train and test our model, contains over 60 minutes of data including 360 degree cylindrical RGB video and 3D pointclouds, in social settings. The presented 3D MOT system demonstrates state-ofthe-art performance against competing methods on the popular 2D tracking KITTI benchmark and serves as a competitive 3D tracking baseline for our dataset and benchmark. Moreover, our tests on our social robot JackRabbot indicate that the system is capable of tracking multiple pedestrians fast and reliably.
Publication title
A Real-Time 3D Multi-Object Tracker and a New Large-Scale Dataset.
Publication authors:
A. Shenoi, M. Patel, J. Gwak, P. Goebel, A. Sadeghian, H. Rezatofighi, R. Martín-Martín and S. Savar
3D multi-object tracking (MOT) is an essential component technology for many real-time applications such as autonomous driving or assistive robotics. Recent work on 3D MOT tend to focus more on developing accurate systems giving less regard to computational cost and system complexity. In contrast, this work proposes a simple yet accurate real-time 3D MOT system. We use an off-the-shelf 3D object detector to obtain oriented 3D bounding boxes from the LiDAR point cloud. Then, a combination of 3D Kalman filter and Hungarian algorithm is used for state estimation and data association. Although our baseline system is a straightforward combination of standard methods, we obtain the state-of-the-art results. To evaluate our baseline system, we propose a new 3D MOT extension to the official KITTI 2D MOT evaluation along with a set of new metrics. Our proposed baseline method for 3D MOT establishes new state-of-the-art performance on 3D MOT for KITTI. Surprisingly, although our baseline system does not use any 2D data as input, we place 2nd on the official KITTI 2D MOT leaderboard. Also, our proposed 3D MOT method runs at a rate of 214.7 FPS, achieving the fastest speed among all modern MOT systems.
Method uses stitched images combined from the individual cameras
Pointcloud
Method uses 3D pointcloud data
Online Tracking
Method does frame-by-frame processing with no lookahead
Offline Tracking
Method does not do in-order frame processing
Public Detections
Method uses publicly available detections
Private Detections
Method uses its own private detections
Evaluation Measures[1]
Measure
Better
Perfect
Description
OSPA(2)IoU
OSPA(2)IoU
lower
0.0
OSPA is a set-based metric which can directly capture a distance, between two
sets of trajectories without a thresholding parameter [2].
OSPA(2)IoU Localization
lower
0.0
Representing different
tracking error such as the displacement and size errors, track ID
switches, track fragmentation or even track late initiation/early
termination [2].
OSPA(2)IoU Cardinality
lower
0.0
Representing
cardinality mismatch between two sets, penalizing missed or
false tracks without an explicit definition for them [2].
HOTA
HOTA
higher
100%
Higher Order Tracking Accuracy [3]. Geometric mean of detection accuracy and association
accuracy. Averaged across localization thresholds.
AssA
higher
100%
Association Accuracy [3]. Association Jaccard index averaged over all matching detections
and then averaged over localization thresholds.
DetA
higher
100%
Detection Accuracy [3]. Detection Jaccard index averaged over localization thresholds.
AssRe
higher
100%
Association Recall [3]. TPA / (TPA + FNA) averaged over all matching detections and then
averaged over localization thresholds.
AssPr
higher
100%
Association Precision [3]. TPA / (TPA + FPA) averaged over all matching detections and then
averaged over localization thresholds.
Localization Accuracy [3]. Average localization similarity averaged over all matching
detections and averaged over localization thresholds.
Clear-MOT
MOTA
higher
100%
Multi-Object Tracking Accuracy (+/- denotes standard deviation across all sequences) [4].
This measure combines three error sources: false positives, missed targets and identity
switches.
MOTP
higher
100%
the total error in estimated position for matched
object-hypothesis pairs over all frames, averaged
by the total number of matches made. [4]
MT
higher
100%
Mostly tracked targets. The ratio of ground-truth trajectories that are covered by a track
hypothesis for at least 80% of their respective life span.
ML
lower
0%
Mostly lost targets. The ratio of ground-truth trajectories that are covered by a track
hypothesis for at most 20% of their respective life span.
Recall
higher
100%
Ratio of correct detections to total number of GT boxes.
Precision
higher
100%
Ratio of TP / (TP+FP).
FP
lower
0
The total number of false positives.
FN
lower
0
The total number of false negatives (missed targets).
ID Switches
lower
0
Number of Identity Switches (ID switch ratio = #ID switches / recall) [4]. Please note that
we follow the stricter definition of identity switches as described in the reference
Identity
IDF1
higher
100%
ID F1 Score [4]. The ratio of correctly identified detections over the average number of
ground-truth and computed detections.
IDP
higher
100%
Identification recall: IDTP/(IDTP+IDFP)[5]
IDR
higher
100%
Identification precision: IDTP/(IDTP+IDFN)[5].
others
-
100%
Referring to the publication[5].
Count
Dets
-
-
Numbers of total detected objects in a sequence.
IDs
-
-
Numbers of total tracked IDs in a sequence.
Reference
The style and content of the Evaluation Measures section is reference from MOT Challenges.
|
|
|
Individual Image
Offline Tracking
