JackRabbot Dataset and Benchmark (JRDB)

Visual Perception for Navigation in Human Environments

Overview

JRDB is a novel dataset collected from our social mobile manipulator, JackRabbot. Our goal is to provide training and benchmarking data for research in the areas of autonomous robot navigation and all perceptual tasks related to social robotics in human environments.

Key Statistics

  • 60,000 annotated frames
  • 2.4 million 2D human bounding box annotations (JRDB)
  • 1.8 million 3D human bounding box annotations (JRDB)
  • 2.8 million individual human action labels (JRDB-Act)
  • 600,000 body pose annotations (JRDB-Pose)
  • 428,000 panoptic segmentation masks and 27,000 mask tracklets (JRDB-PanoTrack)
... and more! Check out the dataset details page for more information.

Download

Please note that you are required to log in before downloading JRDB. If you don't have an account, you can sign up for one here.

See Downloads
Submit to Benchmark
...

The JackRabbot social mobile manipulator.

News and Announcements

May 6, 2026

We have released the JRDB-Pose3D dataset for multi-person 3D human pose and shape estimation in robotics. Paper on arXiv.

November 8, 2025

JRDB-Reasoning has been accepted to AAAI 2026! JRDB-Reasoning is a difficulty-graded benchmark for visual reasoning in robotics, extending JRDB with human-object interaction and geometric relationship annotations. Paper on arXiv.

September 14, 2025

Test set annotations are now available for download! The research community can now access comprehensive test set annotations to evaluate their models. Download here.

June 17, 2024

JRDB-PanoTrack and JRDB-Social datasets have been released!

November 30, 2023

We will soon be launching our new datasets: JRDB-PanoTrack and JRDB-Social.

March 10, 2023

Our new paper titled "JRDB-Pose: A Large-scale Dataset for Multi-Person Pose Estimation and Tracking" has been accepted to CVPR2023. Find it here

October 10, 2022

We have extended the Pose Estimation Challenge deadline to October 15 AoE (Anywhere on Earth). You can find the submission leaderboard here. We look forward to seeing all of your submissions!

August 26, 2022

2D and 3D test set detections and head bounding boxes for both 2D and stitched images are released. See downloads

August 25, 2022

Visualisation toolkit which supports multiple visualisation settings is published, find it here

August 24, 2022

We excited to launch the challenge leaderboard for JRDB 2022! Check them out the leaderboards here. We're running a challenge for our new dataset, JRDB-Pose, and the winners will get to present their work at our workshop at ECCV 2022.

Jun. 27, 2022

As a part of our ECCV22 workshop, we have released JRDB2022 new annotations including 2D human skeleton pose & head bounding boxes and the improved 2&3D bounding box, action & social grouping. New JRDB22 leaderboards will be launched soon.

March 30, 2022

We will organise a new JRDB workshop in conjunction with ECCV22. Click here for more information

March 2, 2022

The JRDB-ACT paper has been accepted and will presented in CVPR22. Find the paper here

November 26, 2021

JRDB dataset and benchmark has offically moved to Monash University! The previous server at Stanford will shut down soon. All previous submissions have been uploaded by the dataset admins. Please register a new account to make submissions.

Downloads

JRDB 2022 (JRDB, JRDB-Act, JRDB-Pose, JRDB-PanoTrack & JRDB-Social) New!

The newest iteration of JRDB, featuring all of our video scenes and annotations.

Plase log in to view these downloads.

JRDB 2019 (JRDB & JRDB-Act)

Plase log in to view these downloads.

Our Papers

JRDB-Reasoning
JRDB-Reasoning: A Difficulty-Graded Benchmark for Visual Reasoning in Robotics

Simindokht Jahangard, Mehrzad Mohammadi, Yi Shen, Zhixi Cai, Hamid Rezatofighi
AAAI Conference on Artificial Intelligence (AAAI), 2026.

See paper on arXiv

Citation:

@inproceedings{jahangard2026jrdbreasoning,
    title={JRDB-Reasoning: A Difficulty-Graded Benchmark for Visual Reasoning in Robotics},
    author={Jahangard, Simindokht and Mohammadi, Mehrzad and Shen, Yi and Cai, Zhixi and Rezatofighi, Hamid},
    booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
    year={2026}
}
JRDB-Pose3D
JRDB-Pose3D: A Multi-person 3D Human Pose and Shape Estimation Dataset for Robotics

Sandika Biswas, Kian Izadpanah, Hamid Rezatofighi
arXiv, 2026.

See paper on arXiv

Citation:

@misc{biswas2026jrdbpose3dmultiperson3dhuman,
    title={JRDB-Pose3D: A Multi-person 3D Human Pose and Shape Estimation Dataset for Robotics}, 
    author={Sandika Biswas and Kian Izadpanah and Hamid Rezatofighi},
    year={2026},
    eprint={2602.03064},
    archivePrefix={arXiv},
    primaryClass={cs.CV},
    url={https://arxiv.org/abs/2602.03064}, 
}
JRDB-PanoTrack: An Open-world Panoptic Segmentation and Tracking Robotic Dataset in Crowded Human Environments

Duy-Tho Le*, Chenhui Gou*, Stavya Datta, Hengcan Shi, Ian Reid, Jianfei Cai, Hamid Rezatofighi
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.

See paper on arXiv

Citation:

@inproceedings{le2024jrdbpanotrack,
    title={JRDB-PanoTrack: An Open-world Panoptic Segmentation and Tracking Robotic Dataset in Crowded Human Environments},
    author={Le, Duy-Tho and Gou, Chenhui and Datta, Stavya and Shi, Hengcan and Reid, Ian and Cai, Jianfei and Rezatofighi, Hamid},
    booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
    year={2024}
}
JRDB-Social: A Multifaceted Robotic Dataset for Understanding of Context and Dynamics of Human Interactions Within Social Groups

Simindokht Jahangard, Zhixi Cai, Shiki Wen, Hamid Rezatofighi
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.

See paper on arXiv

Citation:

@inproceedings{jahangard2024jrdbsocial,
    title={JRDB-Social: A Multifaceted Robotic Dataset for Understanding of Context and Dynamics of Human Interactions Within Social Groups},
    author={Jahangard, Simindokht and Cai, Zhixi and Wen, Shiki and Rezatofighi, Hamid},
    booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
    year={2024}
}
JRDB-Traj: A Dataset and Benchmark for Trajectory Forecasting in Crowds

Saeed Saadatnejad, Yang Gao, Hamid Rezatofighi, Alexandre Alahi
arXiv, 2023.

See paper on arXiv

Citation:

@misc{saadatnejad2023jrdb,
    title={JRDB-Traj: A Dataset and Benchmark for Trajectory Forecasting in Crowds},
    author={Saeed Saadatnejad, Yang Gao, Hamid Rezatofighi, Alexandre Alahi},
    year={2023},
    eprint={2311.02736},
    archivePrefix={arXiv}
}
JRDB-Pose: A Large-scale Dataset for Multi-Person Pose Estimation and Tracking

Edward Vendrow*, Duy-Tho Le*, Jianfei Cai, Hamid Rezatofighi.
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.

See paper on arXiv

Citation:

@inproceedings{vendrow2023jrdb,
    title={JRDB-Pose: A Large-scale Dataset for Multi-Person Pose Estimation and Tracking},
    author = {Vendrow, Edward and Le, Duy Tho and Cai, Jianfei and Rezatofighi, Hamid},
    booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
    year={2023}
}
JRDB-Act: A Large-scale Dataset for Spatio-temporal Action, Social Group and Activity Detection

Mahsa Ehsanpour, Fatemeh Saleh, Silvio Savarese, Ian Reid, Hamid Rezatofighi.
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.

See paper on arXiv

Citation:

@inproceedings{ehsanpour2022jrdb,
    title={JRDB-Act: A Large-Scale Dataset for Spatio-Temporal Action, Social Group and Activity Detection},
    author={Ehsanpour, Mahsa and Saleh, Fatemeh and Savarese, Silvio and Reid, Ian and Rezatofighi, Hamid},
    booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
    year={2022}
}
JRDB: A Dataset and Benchmark of Egocentric Robot Visual Perception of Humans in Built Environments

Roberto Martín-Martín*, Mihir Patel*, Hamid Rezatofighi*, Abhijeet Shenoi, JunYoung Gwak, Eric Frankel, Amir Sadeghian, Silvio Savarese.
IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2021.

See paper on arXiv

Citation:

@article{martin2021jrdb,
    title={Jrdb: A dataset and benchmark of egocentric robot visual perception of humans in built environments},
    author={Martin-Martin, Roberto and Patel, Mihir and Rezatofighi, Hamid and Shenoi, Abhijeet and Gwak, JunYoung and Frankel, Eric and Sadeghian, Amir and Savarese, Silvio},
    journal={IEEE transactions on pattern analysis and machine intelligence},
    year={2021},
    publisher={IEEE}
}
..
..
..
..