JRDB-Social is designed for human social understanding in human interactions across diverse indoor and outdoor social environments. Recognizing humanity's innate social nature, this dataset endeavours to offer a thorough grasp of human behaviour within varied social contexts. JRDB-Social provides annotations at three discernible levels: (1) individual attributes, (2) intra-group interactions, and (3) social group context.

JRDB-Social Visulization

Social Group Level Context

Salient Scene Content (SSC)
1.gate
2.counter
3.pillar
4.standboard
5.poster
6.desk
7.food-truck
8.fence
9.show-case
10.room
11.garbage-bin
12.stroller
13.elevator
14.buffet-cafeteria
15.trolley
16.forecourt
17.bus
18.robot
19.platform
20.tree
21.crutches
22.stand-pillar
23.screen
24.copy-machine
25.class
26.balcony
27.sofa
28.statue
29.bench
30.baggage
31.shop
32.light-street
33.drink-fountain
    

Body Positions and Content (BPC)
1.floor
2.ground
3.chair
4.sidewalk
5.bike
6.stairs
7.platform
8.sofa
9.grass
10.street
11.crosswalk
12.road
13.scooter
14.skateboard
15.pathway
16.desk
17.balcony
18.bench
    

Annotation Format

For each individual, we added new annotations to a .json file, which includes three levels of information: (1) demographics information, (2) human-to-human interaction, and (3) group information, detailed below:

"demographics_info": [
                {'gender': confidence score},
                {'age': confidence score},
                {'race': confidence score},
                ...
            ],
"H-interaction": [
                {'box_pair': [box1,box2,box3,box4],'inter_labels':{'interaction':confidence score},'pair':'ID_pair1'},
                {'box_pair': [box1,box2,box3,box4],'inter_labels':{'interaction':confidence score},'pair':'ID_pair2'},
                ...
            ],
"group_info": [
                {
                'inter': {'interaction':confidence score},
                'BPC': {'body pose content':confidence score},
                'location_pre': {'preposition':confidence score},
                'SSC': {'salient object':confidence score},
                'aim': {'aim':confidence score},
                'venue': {'venue':confidence score},
                }
                ...
            ]


            

Evaluation Protocol

The evaluation of models in JRDB-Social based on textual descriptions employs traditional metrics like BLEU, ROUGE, and METEOR, but these metrics may lack specificity in capturing key entities. To address this, the evaluation framework shifts towards named entity extraction and classification tasks, which provide a more focused assessment of model performance. Additionally, accuracy and F1 score are used to evaluate interaction labels, offering a balanced evaluation in imbalanced scenarios. This approach aims to overcome the limitations of generic metrics and provide a more comprehensive evaluation of model effectiveness.

Toolkit & Code Samples

The evaluation toolkit for JRDB-Social is released as a part of the JRDB-Social dataset.

The toolkit can be found here:

The code samples can be found here:

Downloads

JRDB-Social dataset can be found here: