Secure Multi-Task Federated Panoptic Perception Algorithm for Connected Autonomous Vehicles

HUANG Xiaoge; CHEN Ming; TANG Yi; LIANG Chengchao; CHEN Qianbin

doi:10.11999/JEIT250749

Volume 48 Issue 6

Jun. 2026

Turn off MathJax

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2026 > 48(6): 2591-2601

HUANG Xiaoge, CHEN Ming, TANG Yi, LIANG Chengchao, CHEN Qianbin. Secure Multi-Task Federated Panoptic Perception Algorithm for Connected Autonomous Vehicles[J]. Journal of Electronics & Information Technology, 2026, 48(6): 2591-2601. doi: 10.11999/JEIT250749

Citation:

HUANG Xiaoge, CHEN Ming, TANG Yi, LIANG Chengchao, CHEN Qianbin. Secure Multi-Task Federated Panoptic Perception Algorithm for Connected Autonomous Vehicles[J]. Journal of Electronics & Information Technology, 2026, 48(6): 2591-2601. doi: 10.11999/JEIT250749

Citation:

PDF( 3867 KB)

Secure Multi-Task Federated Panoptic Perception Algorithm for Connected Autonomous Vehicles

doi: 10.11999/JEIT250749 cstr: 32379.14.JEIT250749

School of Communication and Information Engineering, Chongqing University of Posts and Telecommunications, Chongqing 400065, China

Funds: The National Natural Science Foundation of China (62371082), Guangxi Science and Technology Project (AB24010317), The Science and Technology Project of Chongqing Education Commission (KJZD-K202400606), The Natural Science Foundation of Chongqing (CSTB2023NSCQ-MSX0726, CSTB2023NSCQ-LZX0014)

Received Date: 2025-08-12
Accepted Date: 2026-04-12
Rev Recd Date: 2026-04-12

Available Online: 2026-04-30

Publish Date: 2026-06-15

Abstract

Abstract

Objective Connected autonomous vehicles (CAVs) need to perceive complex driving environments accurately and in real time, including traffic object detection, drivable-area segmentation, and lane-line detection. Existing single-task perception models are difficult to meet these diverse requirements simultaneously, while running multiple models in sequence increases latency and is unsuitable for safety-critical autonomous driving. In addition, the performance of deep learning models depends heavily on large-scale and diverse training data, whereas the data collected by an individual CAV are usually limited and scenario-specific. Federated learning (FL) provides a feasible way to train a more generalized global model without uploading raw vehicle data. However, conventional FL still faces several challenges in Internet of Vehicles scenarios, including malicious CAVs that upload low-quality or poisoned local models, privacy leakage from uploaded model parameters, and the single point of failure caused by centralized aggregation. To address these problems, this paper proposes a Secure Multi-task Federated Panoptic Perception (SMFPP) algorithm for connected autonomous vehicles. The goal is to achieve efficient multi-task driving perception, reliable collaborative model training, malicious-node resistance, privacy-preserving aggregation, and fault tolerance in a unified framework. Methods The proposed SMFPP framework consists of a cloud layer, an edge layer, and a terminal layer. The cloud layer publishes FL tasks and provides global coordination, the edge layer contains multiple road side units (RSUs) responsible for model aggregation, and the terminal layer consists of CAVs that train local models with their own driving-scene data. First, a YOLO-based multi-task panoptic perception model is designed as the local model. Its backbone network extracts visual features from vehicle images, while the spatial pyramid pooling fast module and feature pyramid network perform multi-scale feature fusion. Three task-specific detection heads are constructed to perform traffic object detection, drivable-area segmentation, and lane-line detection in parallel, thereby reducing perception latency and improving real-time applicability. The overall loss function combines the losses of the three tasks, including object detection losses and segmentation-related losses. Second, to defend against poisoning attacks and low-quality updates in FL, a hybrid-scoring-based CAV selection algorithm is proposed. For direct scoring, each CAV evaluates the cosine similarity between its local model and the previous global model, as well as the update age of its local model. For indirect scoring, RSUs select high-reputation CAVs to conduct encrypted cross-validation of other CAVs’ local models, and the model quality is evaluated according to local validation performance. The direct and indirect scores are normalized and used as the input of K-means clustering to select CAVs with high model quality, good timeliness, and strong consistency. The reputation value of each CAV is then updated for subsequent training rounds. Third, to avoid centralized aggregation risks, a Shamir-secret-sharing-based global model aggregation algorithm is developed. Each selected CAV splits its local model parameters into secret shares and sends them to different RSUs. Each RSU aggregates only the received shares and returns an aggregation share to CAVs. When a CAV receives at least the threshold number of aggregation shares, it reconstructs the global model by Lagrange interpolation. In this way, no single RSU can obtain a complete local model, and the system can still work when some RSUs fail. Results and Discussions Experiments are conducted on the BDD100K dataset, which contains diverse driving scenes from urban roads, suburban areas, and highways. In the basic simulation setting, five CAVs participate in FL training, each CAV uses 20% of the dataset to train its local model, and five RSUs serve as edge aggregation nodes. The effectiveness of the hybrid-scoring-based CAV selection algorithm is first verified under different numbers of malicious CAVs. As shown in Fig. 2, the mAP50 values of normal CAVs increase with training rounds and gradually converge, whereas malicious CAVs maintain low and unstable mAP50 values. Fig.3 further shows that a malicious CAV is excluded from global aggregation throughout training, and the global model achieves better convergence when the proposed selection algorithm is used. Compared with FedAvg, FedCS, and FedAsync, the proposed SMFPP framework obtains the best overall performance, achieving 86.4% Recall, 77.7% mAP50, 89.6% mIoU, 82.4% Accuracy, and 27.2% IoU, as reported in Table 1. For traffic object detection, SMFPP achieves 86.4% Recall and 77.7% mAP50, and its mAP50 is higher than those of MultiNet, DLT-Net, Fast R-CNN, YOLOv8n, YOLOP, A-YOLOM, and YOLOMH, as shown in Table 2. For drivable-area segmentation, SMFPP reaches 89.6% mIoU, outperforming MultiNet, DLT-Net, PSPNet, YOLOv8n(seg), YOLOP, A-YOLOM, and YOLOMH by 20.2%, 18.8%, 2.0%, 12.2%, 0.3%, 0.9%, and 0.4%, respectively, as shown in Table 3. For lane-line detection, SMFPP achieves the best Accuracy of 82.4% and IoU of 27.2%, as shown in Table 4. In both daytime and nighttime scenarios, SMFPP performs better than YOLOP on all evaluation metrics, demonstrating its adaptability to different illumination conditions (Table 5). In terms of efficiency, SMFPP contains only 3.45 million parameters and reaches 52.4 fps, indicating that it is lightweight and suitable for edge deployment (Table 6). The scalability experiment shows that, when the number of CAVs increases from 5 to 20, the global model performance improves consistently, with Recall, mAP50, mIoU, Accuracy, and IoU increasing to 88.6%, 79.5%, 90.4%, 85.2%, and 28.4%, respectively (Table 7). Moreover, the Shamir-secret-sharing-based aggregation scheme remains effective as long as at least three of the five RSUs survive. When the number of available RSUs is below the threshold, the global model cannot be correctly recovered; once enough RSUs recover, model training continues normally (Fig. 5). These results demonstrate that SMFPP improves perception accuracy, robustness against malicious CAVs, privacy protection, fault tolerance, and scalability. Conclusions This paper presents SMFPP, a secure multi-task federated panoptic perception algorithm for connected autonomous vehicles. The proposed method integrates a lightweight YOLO-based multi-task perception model with FL to solve the data-island problem among CAVs and to support collaborative training without exposing raw driving data. The hybrid-scoring-based CAV selection algorithm effectively filters malicious or low-quality local models by jointly considering model quality, update timeliness, and model similarity. The Shamir-secret-sharing-based aggregation scheme further protects local model parameters and removes the single point of failure in centralized FL aggregation. Simulation results on the BDD100K dataset verify that SMFPP achieves competitive or superior performance in object detection, drivable-area segmentation, and lane-line detection, while maintaining high inference efficiency and strong robustness. Future work may consider larger-scale heterogeneous vehicular networks, dynamic communication conditions, adaptive threshold settings, and real-world deployment in edge-assisted intelligent transportation systems.
- Panoptic perception,
- Deep learning,
- Federated learning,
- Secret sharing

FullText(HTML)

References(18)

References

[1]	DHINESH KUMAR R and RAMMOHAN A. Revolutionizing intelligent transportation systems with cellular vehicle-to-everything (C-V2X) technology: Current trends, use cases, emerging technologies, standardization bodies, industry analytics and future directions[J]. Vehicular Communications, 2023, 43: 100638. doi: 10.1016/j.vehcom.2023.100638.
[2]	HUANG Xiaoge, YIN Hongbo, CHEN Qianbin, et al. DAG-based swarm learning: A secure asynchronous learning framework for Internet of Vehicles[J]. Digital Communications and Networks, 2024, 10(6): 1611–1621. doi: 10.1016/j.dcan.2023.10.004.
[3]	HUANG Xiaoge, LI Wenjing, LIANG Chengchao, et al. Environment-aware personalized heterogeneous federated distillation for dual-layer blockchain-enabled internet of vehicles[J]. IEEE Transactions on Vehicular Technology, 2025, 74(12): 19552–19567. doi: 10.1109/TVT.2025.3586538.
[4]	潘仲夏, 沈丛麒, 骆汉光, 等. 面向车路协同的地理空间标识网络模态设计与场景应用[J]. 电子与信息学报, 2026, 48(2): 583–596. doi: 10.11999/JEIT250807. PAN Zhongxia, SHEN Congqi, LUO Hanguang, et al. Geospatial Identifier Network Modal Design and Scenario Applications for Vehicle-infrastructure Cooperative Networks[J]. Journal of Electronics & Information Technology, 2026, 48(2): 583–596. doi: 10.11999/JEIT250807.
[5]	GUO Yu, LIU R W, LU Yuxu, et al. Haze visibility enhancement for promoting traffic situational awareness in vision-enabled intelligent transportation[J]. IEEE Transactions on Vehicular Technology, 2023, 72(12): 15421–15435. doi: 10.1109/TVT.2023.3298041.
[6]	GIRSHICK R. Fast R-CNN[C]. IEEE International Conference on Computer Vision, Santiago, Chile, 2015: 1440–1448. doi: 10.1109/ICCV.2015.169.
[7]	TERVEN J, CÓRDOVA-ESPARZA D M, and ROMERO-GONZÁLEZ J A. A comprehensive review of YOLO architectures in computer vision: From YOLOv1 to YOLOv8 and YOLO-NAS[J]. Machine Learning and Knowledge Extraction, 2023, 5(4): 1680–1716. doi: 10.3390/make5040083.
[8]	BADRINARAYANAN V, KENDALL A, and CIPOLLA R. SegNet: A deep convolutional encoder-decoder architecture for image segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(12): 2481–2495. doi: 10.1109/TPAMI.2016.2644615.
[9]	ZHAO Hengshuang, SHI Jianping, QI Xiaojuan, et al. Pyramid scene parsing network[C]. IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, 2017: 6230–6239. doi: 10.1109/CVPR.2017.660.
[10]	PAN Xingang, SHI Jianping, LUO Ping, et al. Spatial as deep: Spatial CNN for traffic scene understanding[C]. The 32th AAAI Conference on Artificial Intelligence, New Orleans, USA, 2018: 7276–7283. doi: 10.1609/aaai.v32i1.12301.
[11]	NEVEN D, DE BRABANDERE B, GEORGOULIS S, et al. Towards end-to-end lane detection: An instance segmentation approach[C]. IEEE Intelligent Vehicles Symposium, Changshu, China, 2018: 286–291. doi: 10.1109/IVS.2018.8500547.
[12]	TEICHMANN M, WEBER M, ZÖLLNER M, et al. Multinet: Real-time joint semantic reasoning for autonomous driving[C]. IEEE Intelligent Vehicles Symposium, Changshu, China, 2018: 1013–1020. doi: 10.1109/IVS.2018.8500504.
[13]	WU Dong, LIAO Manwen, ZHANG Weitian, et al. Correction to: YOLOP: You only look once for panoptic driving perception[J]. Machine Intelligence Research, 2023, 20(6): 952. doi: 10.1007/s11633-023-1452-6.
[14]	DAT V T, BAO N V H, and HUNG P D. HybridNets: End-to-end perception network[J]. Pattern Recognition and Image Analysis, 2025, 35(2): 106–118. doi: 10.1134/S1054661825700038.
[15]	XIAO Sa, HUANG Xiaoge, DENG Xuesong, et al. DB-FL: DAG blockchain-enabled generalized federated dropout learning[J]. Digital Communications and Networks, 2025, 11(3): 886–897. doi: 10.1016/j.dcan.2024.09.005.
[16]	郭显, 王典冬, 冯涛, 等. 基于同态加密的可验证隐私保护联邦学习方案[J]. 电子与信息学报, 2025, 47(4): 1113–1125. doi: 10.11999/JEIT240390. GUO Xian, WANG Diandong, FENG Tao, et al. A verifiable privacy protection federated learning scheme based on homomorphic encryption[J]. Journal of Electronics & Information Technology, 2025, 47(4): 1113–1125. doi: 10.11999/JEIT240390.
[17]	李亚红, 李一婧, 杨小东, 等. 基于同态加密和群签名的可验证联邦学习方案[J]. 电子与信息学报, 2025, 47(3): 758–768. doi: 10.11999/JEIT240796. LI Yahong, LI Yijing, YANG Xiaodong, et al. A verifiable federated learning scheme based on homomorphic encryption and group signature[J]. Journal of Electronics & Information Technology, 2025, 47(3): 758–768. doi: 10.11999/JEIT240796.
[18]	HU Xiaoya, LI Ruiqin, WANG Licheng, et al. A data sharing scheme based on federated learning in IoV[J]. IEEE Transactions on Vehicular Technology, 2023, 72(9): 11644–11656. doi: 10.1109/TVT.2023.3266100.