Advancing the state-of-the-art human understanding across diverse indoor and outdoor social contexts.

JRDB-Social is designed for human social understanding in human interactions across diverse indoor and outdoor social environments. Recognizing humanity's innate social nature, this dataset endeavours to offer a thorough grasp of human behaviour within varied social contexts. JRDB-Social provides annotations at three discernible levels: (1) individual attributes, (2) intra-group interactions, and (3) social group context.

JRDB-Social Visulization


Social Group Level Context

Salient Scene Content (SSC)

1.gate 2.counter 3.pillar 4.standboard 5.poster 6.desk 7.food-truck 8.fence 9.show-case 10.room 11.garbage-bin 12.stroller 13.elevator 14.buffet-cafeteria 15.trolley 16.forecourt 17.bus 18.robot 19.platform 20.tree 21.crutches 22.stand-pillar 23.screen 24.copy-machine 25.class 26.balcony 27.sofa 28.statue 29.bench 30.baggage 31.shop 32.light-street 33.drink-fountain

Body Positions and Content (BPC)

1.floor 2.ground 3.chair 4.sidewalk 5.bike 6.stairs 7.platform 8.sofa 9.grass 10.street 11.crosswalk 12.road 13.scooter 14.skateboard 15.pathway 16.desk 17.balcony 18.bench

Annotation Format

For each individual, we added new annotations to a .json file, which includes three levels of information: (1) demographics information, (2) human-to-human interaction, and (3) group information, detailed below:

"demographics_info": [
                {'gender': confidence score},
                {'age': confidence score},
                {'race': confidence score},
"H-interaction": [
                {'box_pair': [box1,box2,box3,box4],'inter_labels':{'interaction':confidence score},'pair':'ID_pair1'},
                {'box_pair': [box1,box2,box3,box4],'inter_labels':{'interaction':confidence score},'pair':'ID_pair2'},
"group_info": [
                'inter': {'interaction':confidence score},
                'BPC': {'body pose content':confidence score},
                'location_pre': {'preposition':confidence score},
                'SSC': {'salient object':confidence score},
                'aim': {'aim':confidence score},
                'venue': {'venue':confidence score},


Evaluation Protocol

The evaluation of models in JRDB-Social based on textual descriptions employs traditional metrics like BLEU, ROUGE, and METEOR, but these metrics may lack specificity in capturing key entities. To address this, the evaluation framework shifts towards named entity extraction and classification tasks, which provide a more focused assessment of model performance. Additionally, accuracy and F1 score are used to evaluate interaction labels, offering a balanced evaluation in imbalanced scenarios. This approach aims to overcome the limitations of generic metrics and provide a more comprehensive evaluation of model effectiveness.

Toolkit & Code Samples

The evaluation toolkit for JRDB-Social is released as a part of the JRDB-Social dataset.

The toolkit can be found here:

JRDB-Social Toolkit

The code samples can be found here:

Github code


JRDB-Social dataset can be found here:

See Downloads