Personalized Federated Learning with Synergistic Optimization of Hypernetwork Modulation and Reinforcement Learning

ZHANG Zhuolang; YAN Fu; CHEN Yuling

doi:10.11999/JEIT260222

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2026 >

ZHANG Zhuolang, YAN Fu, CHEN Yuling. Personalized Federated Learning with Synergistic Optimization of Hypernetwork Modulation and Reinforcement Learning[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260222

Citation:

ZHANG Zhuolang, YAN Fu, CHEN Yuling. Personalized Federated Learning with Synergistic Optimization of Hypernetwork Modulation and Reinforcement Learning[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260222

Citation:

PDF( 4191 KB)

Personalized Federated Learning with Synergistic Optimization of Hypernetwork Modulation and Reinforcement Learning

doi: 10.11999/JEIT260222 cstr: 32379.14.JEIT260222

ZHANG Zhuolang^{1, 2, 3
,},
YAN Fu^{2, 3
,
,},
CHEN Yuling^{2, 3
,}

1.
School of Mathematics and Statistics, Guizhou University, Guiyang 550025, China
2.
Guizhou Provincial Laboratory of Big Data, Guizhou University, Guiyang 550025, China
3.
State Key Laboratory of Public Big Data, Guizhou University, Guiyang 550025, China

Funds: The National Natural Science Foundation of China (No.62466010), Scientific and Technological Innovation Platform Research Project of Guizhou Province (CXPTXM[2025]024), Major Research Project of Guizhou Provincial Laboratory (QKHPT-SSYS[2026]ZD015), Guizhou Province Computing Power Network Security Protection Technology Innovation Talent Team (Qiankehe Talent CXTD[2025]029)

Accepted Date: 2026-11-02
Rev Recd Date: 2026-11-02

Available Online: 2026-11-14

Abstract

Abstract

Objective Federated learning enables multiple clients to collaboratively train models without sharing raw data, but its performance often degrades when client data are Non-Independent and Identically Distributed (Non-IID). Data heterogeneity may cause client drift, weak global generalization, slow convergence, and increased communication cost. Personalized federated learning aims to preserve shared knowledge while adapting models to local distributions, yet existing methods often rely on fixed personalization strategies or generate large amounts of client-specific parameters. As a result, global knowledge sharing and local adaptation are difficult to balance, especially when client states change during training. To address these problems, a personalized federated learning method, FedHR, is proposed by combining hypernetwork-based parameter modulation with Deep Deterministic Policy Gradient (DDPG). The method is designed to achieve dynamic personalization with limited additional communication and to improve robustness under heterogeneous data. Methods FedHR retains the shared feature-extraction layers of the global model and personalizes only the final fully connected layer. A lightweight hypernetwork takes a client embedding as input and generates client-specific scaling and shifting parameters for the last layer. The client embedding is initialized from the local label distribution and is optimized together with the hypernetwork, allowing it to represent differences in client-specific parameter requirements. This partial modulation strategy preserves global feature representations while reducing the output dimension of the hypernetwork. To avoid using a fixed modulation strength, FedHR introduces a DDPG agent to determine how strongly the hypernetwork-generated parameters should affect each client. The state contains five quantities: the Kullback-Leibler divergence between local and global prediction distributions, the Euclidean and cosine distances between model parameters, local accuracy, and training progress. The action is a continuous modulation strength in the interval [0, 1]. A smaller value keeps the model closer to the global model, while a larger value strengthens local personalization. The reward combines local accuracy improvement with a penalty on excessive deviation from the global model. During each communication round, the server aggregates client models, generates client-specific modulation parameters, and sends them together with the global model (Fig.1, Algorithm 1). Each selected client then constructs its state, obtains the modulation strength from the DDPG agent, forms a personalized initialization, performs local training, and uploads the updated model for the next aggregation. Global Accuracy (GA), Personalized Accuracy (PA), and Total Communication Cost (TCC) are adopted as the main evaluation metrics so that predictive performance and communication efficiency can be examined together. A convergence analysis is further provided under standard smoothness, bounded stochastic-gradient, bounded modulation, and policy-approximation assumptions. The analysis shows that the modulation-induced initialization deviation and client drift remain bounded when the corresponding error terms are bounded. Results and Discussions Experiments are conducted on MNIST, CIFAR-10, CIFAR-100, UCI-HAR, and Shakespeare, covering image classification, sensor-based activity recognition, and sequence prediction. Both Dirichlet and pathological Non-IID partitions are considered for the image datasets, with 50 and 100 clients. FedAvg, FedProx, Ditto, FedKD, FedBN, FedPer, and pFedHN are used as baselines. The results show that FedHR achieves better global and personalized accuracy in most settings (Table 1, Table 2), with clearer advantages under stronger heterogeneity and on more complex tasks. Across the reported image experiments, the global accuracy of FedHR is 0.73%–37.90% higher than that of FedAvg, and its personalized accuracy is 0.11%–11.24% higher than that of the best baseline in each setting. For example, on CIFAR-10 with 100 clients under the pathological partition, the personalized accuracy of FedHR reaches 0.9094, compared with 0.8933 for FedPer and 0.8739 for pFedHN (Table 2). The improvement mainly comes from the separation of shared representation learning and client-specific decision-layer adaptation. The shared backbone retains common knowledge across clients, whereas the hypernetwork adjusts the final decision boundary according to local distributions. DDPG further controls the modulation strength according to the current client state, reducing the risk of insufficient or excessive personalization. FedHR also performs well on UCI-HAR and Shakespeare (Table 3), indicating that the modulation mechanism is not restricted to image-classification tasks. On UCI-HAR, FedHR obtains a global accuracy of 0.9257 and a personalized accuracy of 0.9733, and on Shakespeare the two values are 0.4458 and 0.4377, respectively. The training curves show that FedHR reaches high personalized accuracy in fewer communication rounds and maintains a relatively stable loss trajectory (Fig.2, Fig.3). Although the additional client embedding and modulation parameters slightly increase the communication load of each round, the reduction in required communication rounds leads to a lower total communication cost. Specifically, the per-round parameter count of FedHR is only about 1.74% higher than that of FedAvg, whereas FedHR reaches the preset accuracy thresholds within 5, 4, and 6 rounds on MNIST, CIFAR-10, and CIFAR-100, respectively. Compared with representative baselines, the total communication cost is reduced by 27.0%–98.0% in the reported settings (Table 4). Experiments with different Dirichlet parameters further show that the advantage of FedHR becomes more evident when client distributions differ substantially, whereas the performance gap decreases as the data distributions approach IID (Fig.4, Fig.5). In the CIFAR-100 setting with a Dirichlet parameter larger than 0.5, the personalized accuracy of FedHR becomes lower than that of FedBN and pFedHN, whereas its global accuracy remains the highest among the compared methods (Fig.4(c), Fig.5(c)). This is consistent with the design of FedHR: client-specific modulation is more useful when local decision boundaries differ considerably, while weaker modulation is preferable when the global model already fits most clients. The ablation study shows that removing hypernetwork modulation leads to a clear performance decrease, and fixed-strength modulation is generally inferior to DDPG-based adaptive modulation (Fig.6). On CIFAR-100, fixed modulation strengths exhibit a trade-off between global and personalized accuracy, whereas DDPG-based dynamic modulation improves both, raising the global accuracy from 0.1837 without modulation to 0.2694 and the personalized accuracy from 0.6111 to 0.7492. The learned client embeddings also exhibit distribution-related structures in the t-SNE visualization (Fig.7), supporting their use as conditions for generating personalized parameters. Conclusions FedHR combines hypernetwork modulation with reinforcement learning to dynamically balance global knowledge sharing and local adaptation in personalized federated learning. By modifying only the final fully connected layer, the method keeps the shared feature extractor intact and limits additional parameter overhead. DDPG adjusts the modulation strength according to the current client state instead of using a fixed personalization ratio. Experiments on five datasets show that FedHR improves global and personalized accuracy, reduces communication cost, and remains effective under different levels of data heterogeneity. The method is more suitable for medium-to-high heterogeneity scenarios, while weaker personalization is preferable when client distributions are close to IID. These results show that adaptive parameter modulation can improve personalized federated learning while maintaining communication efficiency. However, only the final fully connected layer is modulated in this study; extending the modulation to more layers and strengthening privacy protection for deployment in sensitive scenarios remain future work.
- Personalized Federated Learning,
- Hypernetwork,
- Reinforcement Learning,
- Parameter Modulation,
- Personalized Model

FullText(HTML)

References(34)

References

[1]	刘淼, 夏雨虹, 赵海涛, 等. 面向6G工业物联网的联邦学习: 从需求、愿景到挑战、机遇[J]. 电子与信息学报, 2024, 46(12): 4335–4353. doi: 10.11999/JEIT240574. LIU Miao, XIA Yuhong, ZHAO Haitao, et al. Federated learning technologies for 6G industrial internet of things: From requirements, vision to challenges, opportunities[J]. Journal of Electronics & Information Technology, 2024, 46(12): 4335–4353. doi: 10.11999/JEIT240574.
[2]	杨志刚, 王卓彤, 吴大鹏, 等. 物联网中带有隐私保护的鲁棒联邦学习研究[J]. 电子与信息学报, 2023, 45(12): 4235–4244. doi: 10.11999/JEIT221193. YANG Zhigang, WANG Zhuotong, WU Dapeng, et al. Research on data heterogeneous robust federated learning with privacy protection in internet of things[J]. Journal of Electronics & Information Technology, 2023, 45(12): 4235–4244. doi: 10.11999/JEIT221193.
[3]	温泽诚, 陈磊. 联邦学习综述[J]. 计算机科学与应用, 2025, 15(3): 102–107. doi: 10.12677/csa.2025.153062. WEN Zecheng and CHEN Lei. A review of federated learning[J]. Computer Science and Application, 2025, 15(3): 102–107. doi: 10.12677/csa.2025.153062.
[4]	TEO Z L, JIN Liyuan, LIIU Nan, et al. Federated machine learning in healthcare: A systematic review on clinical applications and technical architecture[J]. Cell Reports Medicine, 2024, 5(2): 101419. doi: 10.1016/j.xcrm.2024.101419.
[5]	温木奇, 温武少. 边缘计算的安全挑战与解决方法综述[J]. 计算机系统应用, 2024, 33(11): 38–47. doi: 10.15888/j.cnki.csa.009702. WEN Muqi and WEN Wushao. Review on security challenges and solutions to edge computing[J]. Computer Systems and Applications, 2024, 33(11): 38–47. doi: 10.15888/j.cnki.csa.009702.
[6]	杨睿哲, 谢欣儒, 滕颖蕾, 等. 工业物联网中基于信息熵的联邦增量学习算法与优化[J]. 电子与信息学报, 2024, 46(8): 3146–3154. doi: 10.11999/JEIT231240. YANG Ruizhe, XIE Xinru, TENG Yinglei, et al. Entropy-based federated incremental learning and optimization in industrial internet of things[J]. Journal of Electronics & Information Technology, 2024, 46(8): 3146–3154. doi: 10.11999/JEIT231240.
[7]	MCMAHAN B, MOORE E, RAMAGE D, et al. Communication-efficient learning of deep networks from decentralized data[C]. Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, Fort Lauderdale, USA, 2017: 1273–1282.
[8]	何帆, 王勇, 杨静, 等. 面向边缘计算场景的个性化联邦学习综述[J]. 通信学报, 2025, 46(7): 206–225. doi: 10.11959/j.issn.1000-436x.2025131. HE Fan, WANG Yong, YANG Jing, et al. Survey of personalized federated learning for edge computing scenarios[J]. Journal on Communications, 2025, 46(7): 206–225. doi: 10.11959/j.issn.1000-436x.2025131.
[9]	FALLAH A, MOKHTARI A, and OZDAGLAR A E. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach[C]. Proceedings of the 34th International Conference on Neural Information Processing Systems, Vancouver, Canada, 2020: 300.
[10]	LI Tian, HU Shengyuan, BEIRAMI A, et al. Ditto: Fair and robust federated learning through personalization[C]. The 38th International Conference on Machine Learning (ICML), 2021: 6357–6368. (查阅网上资料, 未找到对应的出版地信息, 请确认补充).
[11]	LI Xiaoxiao, JIANG Meirui, ZHANG Xiaofei, et al. FedBN: Federated learning on non-IID features via local batch normalization[C]. The 9th International Conference on Learning Representations (ICLR), Virtual Event, Austria, 2021. (查阅网上资料, 出版地信息不确定, 请确认).
[12]	COLLINS L, HASSANI H, MOKHTARI A, et al. Exploiting shared representations for personalized federated learning[C]. The 38th International Conference on Machine Learning (ICML), 2021: 2089–2099. (查阅网上资料, 未找到对应的出版地信息, 请确认补充).
[13]	LI Qinbin, HE Bingsheng, and SONG D. Model-contrastive federated learning[C]. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, USA, 2021: 10708–10717. doi: 10.1109/CVPR46437.2021.01057.
[14]	LI Daliang and WANG Junpu. FedMD: Heterogenous federated learning via model distillation[C]. Proceedings of the NeurIPS 2019 Workshop on Federated Learning for Data Privacy and Confidentiality, Vancouver, Canada, 2019. (查阅网上资料, 出版地信息不确定, 请确认).
[15]	VENKATESWARAN P, ISAHAGIAN V, MUTHUSAMY V, et al. FedGen: Generalizable federated learning for sequential data[C]. 2023 IEEE 16th International Conference on Cloud Computing (CLOUD), Chicago, USA, 2023: 308–318.
[16]	HA D, DAI A M, and LE Q V. HyperNetworks[C]. The 5th International Conference on Learning Representations (ICLR), Toulon, France, 2017.
[17]	SHAMSIAN A, NAVON A, FETAYA E, et al. Personalized federated learning using hypernetworks[C]. The 38th International Conference on Machine Learning (ICML), 2021: 9489–9502. (查阅网上资料, 未找到对应的出版地信息, 请确认补充).
[18]	MA Xiaosong, ZHANG Jie, GUO Song, et al. Layer-wised model aggregation for personalized federated learning[C]. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, USA, 2022: 10082–10091. doi: 10.1109/CVPR52688.2022.00985.
[19]	LI Hongxia, CAI Zhongyi, WANG Jingya, et al. FedTP: Federated learning by transformer personalization[J]. IEEE Transactions on Neural Networks and Learning Systems, 2024, 35(10): 13426–13440. doi: 10.1109/TNNLS.2023.3269062.
[20]	李阳, 姜毅, 陈帅, 等. 多层超网络聚合的个性化联邦学习算法[J]. 计算机工程, 2026, 52(9): 133–142. doi: 10.19678/j.issn.1000-3428.0070679. LI Yang, JIANG Yi, CHEN Shuai, et al. Personalized federated learning algorithm based on multi-layer hypernetwork aggregation[J]. Computer Engineering, 2026, 52(9): 133–142. doi: 10.19678/j.issn.1000-3428.0070679.
[21]	YANG Ziyuan, XIA Wenjun, LU Zexin, et al. Hypernetwork-based physics-driven personalized federated learning for CT imaging[J]. IEEE Transactions on Neural Networks and Learning Systems, 2025, 36(2): 3136–3150. doi: 10.1109/TNNLS.2023.3338867.
[22]	CHEN Leiming, ZHANG Weishan, DONG Cihao, et al. FedDRL: Trustworthy federated learning model fusion method based on staged reinforcement learning[J]. Computing and Informatics, 2024, 43(1): 1–37. doi: 10.31577/cai_2024_1_1.
[23]	BANG J, WOO S, and LEE J. DRL-empowered joint batch size and weighted aggregation adjustment mechanism for federated learning on non-IID data[J]. ICT Express, 2024, 10(4): 863–870. doi: 10.1016/j.icte.2024.04.011.
[24]	LILLICRAP T P, HUNT J J, PRITZEL A, et al. Continuous control with deep reinforcement learning[C]. Proceedings of the 4th International Conference on Learning Representations (ICLR), San Juan, Puerto Rico, 2016.
[25]	BOTTOU L, CURTIS F E, and NOCEDAL J. Optimization methods for large-scale machine learning[J]. SIAM Review, 2018, 60(2): 223–311. doi: 10.1137/16M1080173.
[26]	VIRMAUX A and SCAMAN K. Lipschitz regularity of deep neural networks: Analysis and efficient estimation[C]. Proceedings of the 32nd International Conference on Neural Information Processing Systems, Montréal, Canada, 2018: 3839–3848.
[27]	FUJIMOTO S, VAN HOOF H, and MEGER D. Addressing function approximation error in actor-critic methods[C]. The 35th International Conference on Machine Learning (ICML), Stockholm, Sweden, 2018: 1582–1591.
[28]	LECUN Y, BOTTOU L, BENGIO Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11): 2278–2324. doi: 10.1109/5.726791.
[29]	KRIZHEVSKY A. Learning multiple layers of features from tiny images[R]. Toronto, Canada: University of Toronto, 2009.
[30]	ANGUITA D, GHIO A, ONETO L, et al. A public domain dataset for human activity recognition using smartphones[C]. The 21st European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN), Bruges, Belgium, 2013: 437–442.
[31]	CALDAS S, DUDDU S M K, WU P, et al. LEAF: A benchmark for federated settings[C]. Workshop on Federated Learning for Data Privacy and Confidentiality at NeurIPS 2019, Vancouver, Canada, 2019. (查阅网上资料, 未找到本条文献信息, 请确认).
[32]	LI Tian, SAHU A K, ZAHEER M, et al. Federated optimization in heterogeneous networks[C]. The 3rd Conference on Machine Learning and Systems (MLSys), Austin, USA, 2020: 429–450. (查阅网上资料, 页码信息不确定, 请确认).
[33]	WU Chuhan, WU Fangzhao, LYU Lingjuan, et al. Communication-efficient federated learning via knowledge distillation[J]. Nature Communications, 2022, 13: 2032. doi: 10.1038/s41467-022-29763-x.
[34]	ARIVAZHAGAN M G, AGGARWAL V, SINGH A K, et al. Federated learning with personalization layers[EB/OL]. https://arxiv.org/abs/1912.00818, 2019.