JRDB Leaderboard

OW Panoptic Segmentation Submissions

Name

OSPA↓

OSPA^K_Thing↓

OSPA^K_Stuff↓

OSPA^U_Thing↓

FC_Clip

0.884

0.922

0.748

0.926

Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, Liang-Chieh Chen Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP in NeurIPS 2023

Details

Name	FC_Clip
Submission Date	2024-04-22 04:29:38+00:00
Abstract	Open-vocabulary segmentation is a challenging task requiring segmenting and recognizing objects from an open set of categories. One way to address this challenge is to leverage multi-modal models, such as CLIP, to provide image and text features in a shared embedding space, which bridges the gap between closed-vocabulary and open-vocabulary recognition. Hence, existing methods often adopt a two-stage framework to tackle the problem, where the inputs first go through a mask generator and then through the CLIP model along with the predicted masks. This process involves extracting features from images multiple times, which can be ineffective and inefficient. By contrast, we propose to build everything into a single-stage framework using a shared Frozen Convolutional CLIP backbone, which not only significantly simplifies the current two-stage pipeline, but also remarkably yields a better accuracy-cost trade-off. The proposed FC-CLIP, benefits from the following observations: the frozen CLIP backbone maintains the ability of open-vocabulary classification and can also serve as a strong mask generator, and the convolutional CLIP generalizes well to a larger input resolution than the one used during contrastive image-text pretraining. When training on COCO panoptic data only and testing in a zero-shot manner, FC-CLIP achieve 26.8 PQ, 16.8 AP, and 34.1 mIoU on ADE20K, 18.2 PQ, 27.9 mIoU on Mapillary Vistas, 44.0 PQ, 26.8 AP, 56.2 mIoU on Cityscapes, outperforming the prior art by +4.2 PQ, +2.4 AP, +4.2 mIoU on ADE20K, +4.0 PQ on Mapillary Vistas and +20.1 PQ on Cityscapes, respectively. Additionally, the training and testing time of FC-CLIP is 7.5x and 6.6x significantly faster than the same prior art, while using 5.9x fewer parameters. FC-CLIP also sets a new state-of-the-art performance across various open-vocabulary semantic segmentation datasets. Code at this https URL
Publication title	Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP	Publication authors:	Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, Liang-Chieh Chen
Publication venue and year	NeurIPS 2023	Publication URL:	https://openreview.net/pdf?id=83LJRUzXWj
Code Language	N/A	Hardware	N/A
Code Website	N/A	Code URL	N/A

Visualisation

Best 3 By OSPA ↓

Generating nvidia-aud-2019-04-18_1, Come Back Later!

Worst 3 By OSPA ↓

Per-sequence Results

Sequence	OSPA	OSPA_CARD	OSPA_LOC	OSPA_KNOWN_THING	OSPA_UNKNOWN_THING	OSPA_STUFF	OSPA_SMALL	OSPA_MEDIUM	OSPA_LARGE
cubberly-auditorium-2019-04-22_1	0.907	0.788	0.12	0.937	1.0	0.775	0.973	0.906	0.599
discovery-walk-2019-02-28_0	0.852	0.733	0.119	0.884	1.0	0.764	0.941	0.829	0.635
discovery-walk-2019-02-28_1	0.867	0.694	0.173	0.945	0.826	0.737	0.963	0.849	0.739
food-trucks-2019-02-12_0	0.893	0.688	0.205	0.928	0.985	0.747	0.962	0.946	0.849
gates-ai-lab-2019-04-17_0	0.906	0.772	0.134	0.913	0.96	0.733	0.972	0.865	0.729
gates-basement-elevators-2019-01-17_0	0.892	0.745	0.147	0.954	0.856	0.666	0.995	0.916	0.749
gates-foyer-2019-01-17_0	0.9	0.748	0.152	0.933	0.908	0.795	0.987	0.926	0.706
gates-to-clark-2019-02-28_0	0.842	0.696	0.146	0.87	0.909	0.784	0.902	0.797	0.668
hewlett-class-2019-01-23_0	0.868	0.773	0.095	0.878	0.964	0.708	0.965	0.886	0.721
hewlett-class-2019-01-23_1	0.845	0.739	0.106	0.886	0.88	0.564	0.945	0.899	0.575
huang-2-2019-01-25_1	0.897	0.751	0.147	0.931	1.0	0.754	0.97	0.759	0.683
huang-intersection-2019-01-22_0	0.864	0.701	0.163	0.936	0.899	0.684	0.944	0.83	0.68
indoor-coupa-cafe-2019-02-06_0	0.927	0.798	0.128	0.948	0.955	0.822	0.948	0.907	0.796
lomita-serra-intersection-2019-01-30_0	0.814	0.661	0.153	0.919	0.945	0.525	0.955	0.762	0.573
meyer-green-2019-03-16_1	0.914	0.795	0.12	0.974	1.0	0.747	0.986	0.888	0.72
nvidia-aud-2019-01-25_0	0.9	0.756	0.143	0.911	0.893	0.868	0.948	0.834	0.724
nvidia-aud-2019-04-18_1	0.787	0.674	0.113	0.828	0.472	0.767	0.992	0.771	0.579
nvidia-aud-2019-04-18_2	0.902	0.685	0.218	0.912	0.96	0.874	0.989	0.866	0.756
outdoor-coupa-cafe-2019-02-06_0	0.897	0.715	0.182	0.929	0.727	0.862	0.984	0.826	0.678
quarry-road-2019-02-28_0	0.932	0.8	0.132	0.958	1.0	0.828	0.972	0.822	0.795
serra-street-2019-01-30_0	0.87	0.736	0.133	0.938	nan	0.715	0.937	0.69	0.69
stlc-111-2019-04-19_1	0.841	0.691	0.15	0.884	0.985	0.589	0.984	0.86	0.666
stlc-111-2019-04-19_2	0.882	0.751	0.131	0.909	0.962	0.681	0.989	0.819	0.653
tressider-2019-03-16_2	0.831	0.627	0.204	0.951	1.0	0.576	0.97	0.841	0.604
tressider-2019-04-26_0	0.948	0.844	0.104	0.947	0.995	0.868	0.986	0.908	0.77
tressider-2019-04-26_1	0.939	0.823	0.116	0.945	0.997	0.854	0.96	0.914	0.902
tressider-2019-04-26_3	0.946	0.811	0.134	0.947	0.994	0.906	0.982	0.929	0.799
COMBINED	0.884	0.741	0.143	0.922	0.926	0.748	0.967	0.853	0.705

Symbol	Description
Individual Image	Method uses individual images from each camera
Stitched Image	Method uses stitched images combined from the individual cameras
Pointcloud	Method uses 3D pointcloud data
Online Tracking	Method does frame-by-frame processing with no lookahead
Offline Tracking	Method does not do in-order frame processing
Public Detections	Method uses publicly available detections
Private Detections	Method uses its own private detections

Measure	Better	Perfect	Description
OSPA⁽²⁾
OSPA	lower	0.0	OSPA is a set-based metric which can directly capture a distance, between two sets of mask tracks without a thresholding parameter [2,3].
OSPA Localization	lower	0.0	Representing prediction error such as the displacement, track ID switches, track fragmentation or even track late initiation/early termination [2,3].
OSPA Cardinality	lower	0.0	Representing cardinality mismatch between two sets, penalizing missed or false tracks without an explicit definition for them [2,3].

Leaderboard

Instructions

OW Panoptic Segmentation Submissions

Additional Information Used

Evaluation Measures [1]

Reference