UAVREL: A Benchmark Dataset for Dynamic Relation Understanding in UAV Videos

LIU Xiaorui; DENG Chubo; HOU Zhongyan; YAN Qiwei; LU Wanxuan; HOU Yingyan; YU Hongfeng; SUN Xian

doi:10.11999/JEIT260221

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2026 >

LIU Xiaorui, DENG Chubo, HOU Zhongyan, YAN Qiwei, LU Wanxuan, HOU Yingyan, YU Hongfeng, SUN Xian. UAVREL: A Benchmark Dataset for Dynamic Relation Understanding in UAV Videos[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260221

Citation:

LIU Xiaorui, DENG Chubo, HOU Zhongyan, YAN Qiwei, LU Wanxuan, HOU Yingyan, YU Hongfeng, SUN Xian. UAVREL: A Benchmark Dataset for Dynamic Relation Understanding in UAV Videos[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260221

Citation:

LIU Xiaorui, DENG Chubo, HOU Zhongyan, YAN Qiwei, LU Wanxuan, HOU Yingyan, YU Hongfeng, SUN Xian. UAVREL: A Benchmark Dataset for Dynamic Relation Understanding in UAV Videos[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260221

PDF( 7648 KB)

UAVREL: A Benchmark Dataset for Dynamic Relation Understanding in UAV Videos

doi: 10.11999/JEIT260221 cstr: 32379.14.JEIT260221

LIU Xiaorui^{1, 2, 3, 4},
DENG Chubo^{1, 4
,
,},
HOU Zhongyan^{1, 2, 3, 4},
YAN Qiwei^{1, 2, 3, 4},
LU Wanxuan^{1, 4},
HOU Yingyan^{1, 2, 3, 4},
YU Hongfeng^{1, 4},
SUN Xian^{1, 2, 3, 4}

1.
Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100190, China
2.
University of Chinese Academy of Sciences, Beijing 100190, China
3.
School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences, Beijing 100049, China
4.
Key Laboratory of Target Cognition and Application Technology(TCAT), Beijing 100190, China

Funds: The National Natural Science Foundation of China (42301437)

Received Date: 2026-03-01
Accepted Date: 2026-07-28
Rev Recd Date: 2026-07-28

Available Online: 2026-09-01

Abstract

Abstract

Objective With the rapid development and extensive application of Unmanned Aerial Vehicle (UAV) observation platforms, remote sensing video data with high spatiotemporal resolution are increasing rapidly. Dynamic relation understanding in UAV videos is a challenging task in intelligent remote sensing analysis. Current research on video scene graph generation focuses mainly on natural videos, whereas studies on remote sensing videos remain at an early stage. Benchmark datasets with high-order dynamic relation annotations are also lacking. Conventional visual models are difficult to apply directly to remote sensing scenes, which are characterized by substantial viewpoint changes, large variations in object scale, and dense object distributions. To address these limitations, a UAV Video Relation Dataset (UAVREL) with high-order dynamic relation annotations is constructed. A spatiotemporal hypergraph-enhanced relation understanding method tailored to UAV videos is also proposed. In addition, a unified benchmark evaluation system for video scene graph generation in UAV remote sensing is established. These efforts facilitate the transition of intelligent remote sensing interpretation from static object recognition to global dynamic scene understanding. Methods This study consists of three main components: dataset construction, model design, and comprehensive experimental evaluation. First, the Unmanned Aerial Vehicle Benchmark: Object Detection and Tracking (UAVDT) dataset is selected as the basic data source. Fifty representative UAV videos are finely annotated with 12 object categories, resulting in 876,687 object bounding boxes. Twenty relation categories are further annotated based on the object bounding boxes, resulting in 459,164 relation triplets. A semiautomatic annotation strategy integrating object tracking, multi-person collaboration, and cross-checking is developed to reduce subjective bias and improve annotation efficiency. Relations are divided into three levels according to the number of frames required for accurate determination, with high-order dynamic relations prioritized during annotation. Second, a spatiotemporal hypergraph-enhanced relation understanding method (STHG) is proposed. The method comprises six core modules: basic feature extraction, pairwise context encoding, spatiotemporal dual-hypergraph enhancement, multi-source feature fusion, global temporal modeling, and relation prediction. A multi-label classifier is then used for final relation prediction. In the experimental evaluation, an object detection task and three video scene graph generation subtasks, namely Predicate Classification (PredCls), Scene Graph Classification (SGCls), and Scene Graph Detection (SGDet), are conducted on the UAVREL dataset. Representative object detection models and video scene graph generation methods are selected as baselines. Mean Average Precision (mAP) and mAP@50 are used to evaluate object detection, whereas Recall@K (R@K) and mean Recall@K (mR@K) are used to evaluate video scene graph generation. Results and Discussions In the object detection task, eight models with different architectures are compared to evaluate their detection performance (Table 2). The results show that model architecture affects mAP, a core detection metric. Single-stage anchor-free models represented by VFNet and DDOD achieve the best overall detection performance. All models perform poorly on small and easily occluded object categories, such as pedestrians and non-motorized vehicles. In the video scene graph generation task, five methods are evaluated on three subtasks (Tables 3–5). STHG achieves improved performance in SGDet, with R@20, R@50, and R@100 increasing by 2.2%, 1.2%, and 0.4%, respectively, compared with the STTran baseline. In PredCls, STHG performs comparably to STTran, whereas its mR@100 in SGCls is higher than that of STTran. In SGDet, STHG improves mR@20, mR@50, and mR@100 by 4.6%, 4.2%, and 2.2%, respectively, over STTran. The mR@K values are consistently lower than the corresponding R@K values, indicating substantial challenges in modeling long-tailed relations. These results also demonstrate the difficulty of relation prediction in complex UAV video scenes. Conclusions This study focuses on dynamic relation understanding in UAV videos. The UAVREL benchmark dataset is constructed to provide data support for high-order dynamic relation understanding in UAV videos. A spatiotemporal hypergraph-enhanced relation understanding method is also proposed. Experimental results demonstrate its effectiveness across multiple evaluation metrics, particularly for SGDet and long-tailed relation recognition. The dataset and method provide a basis for advancing remote sensing video analysis from static object recognition to high-level dynamic scene understanding. Future research will focus on two directions: improving temporal modeling for dynamic remote sensing scene graph generation to capture long-term evolution and complex relations, and expanding the scene coverage and diversity of relation categories in the dataset to support further development and application of scene graph generation methods.
- Video scene graph generation,
- Remote sensing,
- Relation understanding

FullText(HTML)

References(41)

References

[1]	LIU Fang, WANG Jiahao, JIAO Licheng, et al. Remote sensing video tracking: Current status, challenges, and future[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2025, 18: 14338–14367. doi: 10.1109/JSTARS.2025.3573572.
[2]	MA Bokun, MU Caihong, LIU Yi, et al. RoSENet: Rotation and similarity enhancement network for multimodal remote sensing image land cover classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2025, 63: 5511318. doi: 10.1109/TGRS.2025.3561850.
[3]	金晶, 王峰. 分布式多卫星协同遥感图像场景分类方法[J]. 电子与信息学报, 2025, 47(12): 4677–4688. doi: 10.11999/JEIT250866. JIN Jing and WANG Feng. A distributed multi-satellite collaborative framework for remote sensing scene classification[J]. Journal of Electronics & Information Technology, 2025, 47(12): 4677–4688. doi: 10.11999/JEIT250866.
[4]	GAO Feng, JIN Xuepeng, ZHOU Xiaowei, et al. MSFMamba: Multiscale feature fusion state space model for multisource remote sensing image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2025, 63: 5504116. doi: 10.1109/TGRS.2025.3535622.
[5]	ZHANG Yin, YE Mu, ZHU Guiyi, et al. FFCA-YOLO for small object detection in remote sensing images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2024, 62: 5611215. doi: 10.1109/TGRS.2024.3363057.
[6]	XIAO Yao, XU Tingfa, YU Xin, et al. A lightweight fusion strategy with enhanced interlayer feature correlation for small object detection[J]. IEEE Transactions on Geoscience and Remote Sensing, 2024, 62: 4708011. doi: 10.1109/TGRS.2024.3457155.
[7]	CHEN Tianxiang, YE Zi, TAN Zhentao, et al. MiM-ISTD: Mamba-in-mamba for efficient infrared small-target detection[J]. IEEE Transactions on Geoscience and Remote Sensing, 2024, 62: 5007613. doi: 10.1109/TGRS.2024.3485721.
[8]	姚婷婷, 肇恒鑫, 冯子豪, 等. 上下文感知多感受野融合网络的定向遥感目标检测[J]. 电子与信息学报, 2025, 47(1): 233–243. doi: 10.11999/JEIT240560. YAO Tingting, ZHAO Hengxin, FENG Zihao, et al. A context-aware multiple receptive field fusion network for oriented object detection in remote sensing images[J]. Journal of Electronics & Information Technology, 2025, 47(1): 233–243. doi: 10.11999/JEIT240560.
[9]	SUN Xian, WANG Peijin, YAN Zhiyuan, et al. FAIR1M: A benchmark dataset for fine-grained object recognition in high-resolution remote sensing imagery[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2022, 184: 116–130. doi: 10.1016/j.isprsjprs.2021.12.004.
[10]	周国宇, 张菁, 闫伊, 等. 聚焦注意力与紧致特征融合Transformer的城市遥感影像语义分割[J]. 电子与信息学报, 2025, 47(12): 4790–4800. doi: 10.11999/JEIT250812. ZHOU Guoyu, ZHANG Jing, YAN Yi, et al. A focused attention and feature compact fusion Transformer for semantic segmentation of urban remote sensing images[J]. Journal of Electronics & Information Technology, 2025, 47(12): 4790–4800. doi: 10.11999/JEIT250812.
[11]	CHEN Keyan, LIU Chenyang, CHEN Hao, et al. RSPrompter: Learning to prompt for remote sensing instance segmentation based on visual foundation model[J]. IEEE Transactions on Geoscience and Remote Sensing, 2024, 62: 4701117. doi: 10.1109/TGRS.2024.3356074.
[12]	于国栋, 蒋一纯, 刘云清, 等. 一种空间语义联合感知的红外无人机目标跟踪方法[J]. 电子与信息学报, 2025, 47(11): 4242–4253. doi: 10.11999/JEIT250613. YU Guodong, JIANG Yichun, LIU Yunqing, et al. A spatial-semantic combine perception for infrared UAV target tracking[J]. Journal of Electronics & Information Technology, 2025, 47(11): 4242–4253. doi: 10.11999/JEIT250613.
[13]	JI Jingwei, KRISHNA R, LI Feifei, et al. Action genome: Actions as compositions of spatio-temporal scene graphs[C]. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, USA, 2020: 10233–10244. doi: 10.1109/CVPR42600.2020.01025.
[14]	SADEGHI M A and FARHADI A. Recognition using visual phrases[C]. The 2011 IEEE Conference on Computer Vision and Pattern Recogniti, Colorado Springs, USA, 2011: 1745–1752. doi: 10.1109/CVPR.2011.5995711.
[15]	LU Cewu, KRISHNA R, BERNSTEIN M, et al. Visual relationship detection with language priors[C]. 14th European Conference Computer Vision-ECCV 2016, Amsterdam, The Netherlands, 2016: 852–869. doi: 10.1007/978-3-319-46448-0_51.
[16]	KRISHNA R, ZHU Yuke, GROTH O, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations[J]. International Journal of Computer Vision, 2017, 123(1): 32–73. doi: 10.1007/s11263-016-0981-7.
[17]	LIANG Yuanzhi, BAI Yalong, ZHANG Wei, et al. VrR-VG: Refocusing visually-relevant relationships[C]. 2019 IEEE/CVF International Conference on Computer Vision, Seoul, Korea, 2019: 10402–10411. doi: 10.1109/ICCV.2019.01050.
[18]	SHANG Xindi, REN Tongwei, GUO Jingfan, et al. Video visual relation detection[C]. The 25th ACM International Conference on Multimedia, Mountain View, USA, 2017: 1300–1308. doi: 10.1145/3123266.3123380.
[19]	SHANG Xindi, DI Donglin, XIAO Junbin, et al. Annotating objects and relations in user-generated videos[C]. The 2019 on International Conference on Multimedia Retrieval, Ottawa, Canada, 2019: 279–287.
[20]	YANG Jingkang, PENG Wenxuan, LI Xiangtai, et al. Panoptic video scene graph generation[C]. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, Canada, 2023: 18675–18685. doi: 10.1109/CVPR52729.2023.01791.
[21]	XIA Guisong, BAI Xiang, DING Jian, et al. DOTA: A large-scale dataset for object detection in aerial images[C]. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018: 3974–3983. doi: 10.1109/CVPR.2018.00418.
[22]	CHENG Gong, WANG Jiabao, LI Ke, et al. Anchor-free oriented proposal generator for object detection[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 5625411. doi: 10.1109/TGRS.2022.3183022.
[23]	RIZ L, CARAFFA A, BORTOLON M, et al. The MONET dataset: Multimodal drone thermal dataset recorded in rural scenarios[C]. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Vancouver, Canada, 2023: 2546–2554. doi: 10.1109/CVPRW59228.2023.00253.
[24]	YAN Qiwei, DENG Chubo, LIU Chenglong, et al. ReCon1M: A large-scale benchmark dataset for relation comprehension in remote sensing imagery[J]. IEEE Transactions on Geoscience and Remote Sensing, 2025, 63: 4507022. doi: 10.1109/TGRS.2025.3589986.
[25]	DU Dawei, QI Yuankai, YU Hongyang, et al. The unmanned aerial vehicle benchmark: Object detection and tracking[C]. 15th European Conference Computer Vision-ECCV 2018, Munich, Germany, 2018: 370–386. doi: 10.1007/978-3-030-01249-6_23.
[26]	LYU Ye, VOSSELMAN G, XIA Guisong, et al. UAVid: A semantic segmentation dataset for UAV imagery[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 165: 108–119. doi: 10.1016/j.isprsjprs.2020.05.009.
[27]	HAMDI S, BOUINDOUR S, SNOUSSI H, et al. End-to-end deep one-class learning for anomaly detection in UAV video stream[J]. Journal of Imaging, 2021, 7(5): 90. doi: 10.3390/jimaging7050090.
[28]	TRAN T M, VU T N, NGUYEN T V, et al. UIT-ADrone: A novel drone dataset for traffic anomaly detection[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2023, 16: 5590–5601. doi: 10.1109/JSTARS.2023.3285905.
[29]	NGUYEN T T, NGUYEN P, LI Xin, et al. CYCLO: Cyclic graph transformer approach to multi-object relationship modeling in aerial videos[C]. The 38th International Conference on Neural Information Processing Systems, Vancouver, Canada, 2024: 2868.
[30]	REN Shaoqing, HE Kaiming, GIRSHICK R, et al. Faster R-CNN: Towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1137–1149. doi: 10.1109/tpami.2016.2577031.
[31]	CAI Zhaowei and VASCONCELOS N. Cascade R-CNN: Delving into high quality object detection[C]. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018: 6154–6162. doi: 10.1109/CVPR.2018.00644.
[32]	LIN T Y, GOYAL P, GIRSHICK R, et al. Focal loss for dense object detection[C]. 2017 IEEE International Conference on Computer Vision, Venice, Italy, 2017: 2999–3007. doi: 10.1109/ICCV.2017.324.
[33]	ZHANG Shifeng, CHI Cheng, YAO Yongqiang, et al. Bridging the gap between anchor-based and anchor-free detection via adaptive training sample selection[C]. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, USA, 2020: 9756–9765. doi: 10.1109/CVPR42600.2020.00978.
[34]	ZHU Benjin, WANG Jianfeng, JIANG Zhengkai, et al. AutoAssign: Differentiable label assignment for dense object detection[EB/OL]. https://arxiv.org/abs/2007.03496, 2020.
[35]	TIAN Zhi, SHEN Chunhua, CHEN Hao, et al. FCOS: A simple and strong anchor-free object detector[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(4): 1922–1933. doi: 10.1109/tpami.2020.3032166.
[36]	ZHANG Haoyang, WANG Ying, DAYOUB F, et al. VarifocalNET: An IoU-aware dense object detector[C]. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, USA, 2021: 8501–8519. doi: 10.1109/CVPR46437.2021.00841.
[37]	CHEN Zehui, YANG Chenhongyi, LI Qiaofei, et al. Disentangle your dense object detector[C]. The 29th ACM International Conference on Multimedia, 2021: 4939–4948. doi: 10.1145/3474085.3475351.
[38]	TANG Kaihua, NIU Yulei, HUANG Jianqiang, et al. Unbiased scene graph generation from biased training[C]. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, USA, 2020: 3713–3722. doi: 10.1109/CVPR42600.2020.00377.
[39]	ZELLERS R, YATSKAR M, THOMSON S, et al. Neural motifs: Scene graph parsing with global context[C]. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018: 5831–5840. doi: 10.1109/CVPR.2018.00611.
[40]	TANG Kaihua, ZHANG Hanwang, WU Baoyuan, et al. Learning to compose dynamic tree structures for visual contexts[C]. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, USA, 2019: 6612–6621. doi: 10.1109/CVPR.2019.00678.
[41]	CONG Yuren, LIAO Wentong, ACKERMANN H, et al. Spatial-temporal transformer for dynamic scene graph generation[C]. 2021 IEEE/CVF International Conference on Computer Vision, Montreal, Canada, 2021: 16352–16362. doi: 10.1109/ICCV48922.2021.01606.