高级搜索

留言板

尊敬的读者、作者、审稿人, 关于本刊的投稿、审稿、编辑和出版的任何问题, 您可以本页添加留言。我们将尽快给您答复。谢谢您的支持!

姓名
邮箱
手机号码
标题
留言内容
验证码

超网络调制与强化学习协同优化的个性化联邦学习

张卓琅,  晏福,  陈玉玲

张卓琅, 晏福, 陈玉玲. 超网络调制与强化学习协同优化的个性化联邦学习[J]. 电子与信息学报. doi: 10.11999/JEIT260222
引用本文: 张卓琅, 晏福, 陈玉玲. 超网络调制与强化学习协同优化的个性化联邦学习[J]. 电子与信息学报. doi: 10.11999/JEIT260222
ZHANG Zhuolang, YAN Fu, CHEN Yuling. Personalized Federated Learning with Synergistic Optimization of Hypernetwork Modulation and Reinforcement Learning[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260222
Citation: ZHANG Zhuolang, YAN Fu, CHEN Yuling. Personalized Federated Learning with Synergistic Optimization of Hypernetwork Modulation and Reinforcement Learning[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260222

超网络调制与强化学习协同优化的个性化联邦学习

doi: 10.11999/JEIT260222 cstr: 32379.14.JEIT260222
基金项目: 国家自然科学基金(No.62466010),贵州省科技创新平台科研项目(黔科合平台CXPTXM[2025]024),贵州省实验室重大科研项目(黔科合平台SSYS[2026]重大015),贵州省算力网络安全防护科技创新人才团队(黔科合人才CXTD[2025]029)
详细信息
    作者简介:

    张卓琅:男,硕士生,研究方向为联邦学习,邮箱: 13385199840@163.com

    晏福:男,副教授,硕士研究生导师,研究方向为优化算法、联邦学习,邮箱: yanfu@gzu.edu.cn

    陈玉玲:女,教授,博士研究生导师,研究方向为网络空间安全,邮箱: ylchen3@gzu.edu.cn

    通讯作者:

    晏福 yanfu@gzu.edu.cn

  • 中图分类号: TP391

Personalized Federated Learning with Synergistic Optimization of Hypernetwork Modulation and Reinforcement Learning

Funds: The National Natural Science Foundation of China (No.62466010), Scientific and Technological Innovation Platform Research Project of Guizhou Province (CXPTXM[2025]024), Major Research Project of Guizhou Provincial Laboratory (QKHPT-SSYS[2026]ZD015), Guizhou Province Computing Power Network Security Protection Technology Innovation Talent Team (Qiankehe Talent CXTD[2025]029)
  • 摘要: 联邦学习解决了传统集中式机器学习面临的隐私泄露与“数据孤岛”难题。然而联邦学习参与客户端的数据异质性会导致客户端模型漂移、泛化误差大、全局与局部知识融合低效等局限。针对以上问题,该文提出一种基于超网络参数调制与强化学习自适应融合的个性化联邦学习算法FedHR。FedHR利用超网络为联邦学习每个客户端生成专属调制参数,在保留全局通用知识的基础上嵌入本地数据分布特性,实现全局知识与个性化知识的有效融合。同时,引入深度确定性策略梯度强化学习机制,根据客户端实时训练状态动态分配调制权重,自适应地平衡全局一致性与本地个性化需求。实验结果表明,FedHR的全局与个性化模型准确率显著优于经典联邦学习基准算法,降低了总通信成本,且在异质性场景中展现出较强鲁棒性;消融实验进一步验证了强化学习动态调制的有效性。该文提出的FedHR算法为隐私保护场景下的分布式训练提供了通信高效、预测性能优异且能较好适配复杂异构环境的个性化解决方案。
  • 图  1  FedHR算法整体框架图

    图  2  不同数据集上个性化准确率曲线图

    图  3  不同数据集上个性化损失曲线图

    图  4  不同异质性下个性化模型的最佳准确率

    图  5  不同异质性下全局模型的最佳准确率

    图  6  各调制策略在不同数据集下的最佳准确率

    图  7  客户端嵌入向量可视化

    1  FedHR

     输入:全局通信轮次上限$ T $,客户端本地训练轮次$ E $,预初始化
     全局模型$ \boldsymbol{\theta }_{0}^{\text{global}} $,超网络$ \mathcal{H}({\boldsymbol{\psi }}_{0}) $,强化学习智能体初始化参数。
     输出:最终全局模型参数$ \boldsymbol{\theta }_{T}^{\text{global}} $,各客户端个性化模型参数
     $ \left\{\boldsymbol{\theta }_{T}^{(k)}\right\}_{k=1}^{K} $。
     (1) FOR$ t=1 $TO$ T $DO:
     (2)  根据目标模型最后一个线性层计算调制参数维度$ {\omega }_{\text{size}},{b}_{\text{size}} $
     (3)  输入客户端嵌入$ {\boldsymbol{v}}_{k} $至超网络$ \mathcal{H},\varPhi_{t}^{\left(k\right)}=\mathcal{H}({\boldsymbol{v}}_{k}) $
     (4)  随机采样客户端子集$ {S}_{t} $,向$ {S}_{t} $中每个客户端发送$ \boldsymbol{\theta }_{t}^{\text{global}} $和
     $ \varPhi_{t}^{(k)} $
     (5)  FOR每个$ k\in {S}_{t} $ DO:
     (6)   客户端$ k $接收$ \boldsymbol{\theta }_{t}^{\text{global}} $和$ \varPhi_{t}^{(k)} $
     (7)   构建DDPG状态: $ \boldsymbol{s}_{t}^{(k)} = [{{d_{\text{KL}}^{t}}}^{(k)},{{d_{\text{eu}}^{t}}}^{(k)},{{d_{\cos }^{t}}}^{(k)}, $
     $\text{acc}_{t}^{(k)},{r}_{t}] $
     (8)   Actor网络输出动作:$ \alpha _{t}^{(k)}={\pi }_{\boldsymbol{\phi }}(\boldsymbol{s}_{t}^{(k)}) $
     (9)   调制全局模型最后一层参数:
     $ \boldsymbol{W}\leftarrow \left(1+{{\boldsymbol{\gamma }_{\omega }^{t}}}^{(k)}\cdot \alpha _{t}^{(k)}\right)\odot \boldsymbol{W}+{{\boldsymbol{\beta }_{\omega }^{t}}}^{(k)}\cdot \alpha _{t}^{(k)} $
     $ \boldsymbol{b}\leftarrow \left(1+{{\boldsymbol{\gamma }_{\text{b}}^{t}}}^{(k)}\cdot \alpha _{t}^{(k)}\right)\odot \boldsymbol{b}+{{\boldsymbol{\beta }_{\text{b}}^{t}}}^{(k)}\cdot \alpha _{t}^{(k)} $
     (10) 客户端本地模型训练得到客户端个性化模型参数$ \boldsymbol{\theta }_{t}^{(k)} $
     (11) 根据式(13)计算奖励$ R_{t}^{(k)} $,构建下一状态$ \boldsymbol{s}_{t+1}^{(k)} $,并更新
     Critic与Actor网络
     (12) 服务器聚合模型并更新得到$ \boldsymbol{\theta }_{t+1}^{\text{global}} $
     (13) RETURN最终全局模型参数$ \boldsymbol{\theta }_{T}^{\text{global}} $,各客户端个性化模型
     参数$ \left\{\boldsymbol{\theta }_{T}^{(k)}\right\}_{k=1}^{K} $
    下载: 导出CSV

    表  1  FedHR算法和对比算法的全局模型最佳准确率


    对比
    算法
    MNISTCIFAR-10CIFAR-100
    50个客户端100个客户端50个客户端100个客户端50个客户端100个客户端
    狄利
    克雷
    病态
    划分
    狄利
    克雷
    病态
    划分
    狄利
    克雷
    病态
    划分
    狄利
    克雷
    病态
    划分
    狄利
    克雷
    病态
    划分
    狄利
    克雷
    病态
    划分
    FedAvg0.98500.95570.98170.95970.64110.55400.60380.55580.25310.25000.18630.2361
    FedProx0.98520.95190.98240.95650.64070.55820.60490.55240.25470.24860.17750.2527
    Ditto0.98480.95510.98340.95600.64250.55640.61420.55730.25030.25340.19030.2452
    FedHR0.99220.99010.99300.99020.64820.62370.63340.60950.27250.25940.25690.2508
    下载: 导出CSV

    表  2  FedHR算法和对比算法的个性化模型最佳准确率


    对比
    算法
    MNISTCIFAR-10CIFAR-100
    50个客户端100个客户端50个客户端100个客户端50个客户端100个客户端
    狄利
    克雷
    病态
    划分
    狄利
    克雷
    病态
    划分
    狄利
    克雷
    病态
    划分
    狄利
    克雷
    病态
    划分
    狄利
    克雷
    病态
    划分
    狄利
    克雷
    病态
    划分
    FedAvg0.99080.98720.98850.98790.77240.82570.72600.81750.31990.55720.24450.5579
    FedProx0.99140.98730.98850.98890.77640.83670.72900.81490.31910.54500.24930.5201
    Ditto0.97510.99530.96390.99520.73570.87540.70260.85460.25710.57070.22160.5266
    FedKD0.98820.99780.97840.99690.77150.88790.73160.86590.31680.59690.26640.5645
    FedBN0.98460.95310.98240.96040.63590.55680.60120.55020.29870.25320.27590.2438
    FedPer0.98760.99770.97950.99780.78100.89990.73440.89330.28570.59480.23150.5613
    pFedHN0.98360.99520.98650.99410.49730.87910.50070.87390.22850.58150.22260.4973
    FedHR0.99510.99900.99510.99890.80560.91780.78510.90940.33090.60700.30690.5834
    下载: 导出CSV

    表  3  跨任务模型最佳准确率对比

    对比
    算法
    全局模型个性化模型
    HARShakespeareHARShakespeare
    FedAvg0.86180.27460.93000.2680
    FedProx0.86730.26700.94080.2703
    Ditto0.87040.27370.92570.1903
    FedKD--0.95430.1739
    FedBN--0.86110.2684
    FedPer--0.96320.2068
    pFedHN--0.94470.2351
    FedHR0.92570.44580.97330.4377
    下载: 导出CSV

    表  4  通信成本对比

    对比算法单轮参数量(个)MNISTCIFAR-10CIFAR-100
    达到阈值轮次总通信成本(MB)达到阈值轮次总通信成本(MB)达到阈值轮次总通信成本(MB)
    FedAvg35,503,58613818,690.3820027,047.06729,736.94
    FedProx35,503,58615621,124.41--19125,876.58
    Ditto35,503,586273,650.83476,359.4010614,320.70
    FedKD33,749,716131,669.73182,311.04253,209.72
    FedBN35,503,586------
    FedPer35,190,6567939.58121,609.8322229,828.13
    pFedHN35,503,586------
    FedHR36,121,0265686.034548.826823.24
    下载: 导出CSV
  • [1] 刘淼, 夏雨虹, 赵海涛, 等. 面向6G工业物联网的联邦学习: 从需求、愿景到挑战、机遇[J]. 电子与信息学报, 2024, 46(12): 4335–4353. doi: 10.11999/JEIT240574.

    LIU Miao, XIA Yuhong, ZHAO Haitao, et al. Federated learning technologies for 6G industrial internet of things: From requirements, vision to challenges, opportunities[J]. Journal of Electronics & Information Technology, 2024, 46(12): 4335–4353. doi: 10.11999/JEIT240574.
    [2] 杨志刚, 王卓彤, 吴大鹏, 等. 物联网中带有隐私保护的鲁棒联邦学习研究[J]. 电子与信息学报, 2023, 45(12): 4235–4244. doi: 10.11999/JEIT221193.

    YANG Zhigang, WANG Zhuotong, WU Dapeng, et al. Research on data heterogeneous robust federated learning with privacy protection in internet of things[J]. Journal of Electronics & Information Technology, 2023, 45(12): 4235–4244. doi: 10.11999/JEIT221193.
    [3] 温泽诚, 陈磊. 联邦学习综述[J]. 计算机科学与应用, 2025, 15(3): 102–107. doi: 10.12677/csa.2025.153062.

    WEN Zecheng and CHEN Lei. A review of federated learning[J]. Computer Science and Application, 2025, 15(3): 102–107. doi: 10.12677/csa.2025.153062.
    [4] TEO Z L, JIN Liyuan, LIIU Nan, et al. Federated machine learning in healthcare: A systematic review on clinical applications and technical architecture[J]. Cell Reports Medicine, 2024, 5(2): 101419. doi: 10.1016/j.xcrm.2024.101419.
    [5] 温木奇, 温武少. 边缘计算的安全挑战与解决方法综述[J]. 计算机系统应用, 2024, 33(11): 38–47. doi: 10.15888/j.cnki.csa.009702.

    WEN Muqi and WEN Wushao. Review on security challenges and solutions to edge computing[J]. Computer Systems and Applications, 2024, 33(11): 38–47. doi: 10.15888/j.cnki.csa.009702.
    [6] 杨睿哲, 谢欣儒, 滕颖蕾, 等. 工业物联网中基于信息熵的联邦增量学习算法与优化[J]. 电子与信息学报, 2024, 46(8): 3146–3154. doi: 10.11999/JEIT231240.

    YANG Ruizhe, XIE Xinru, TENG Yinglei, et al. Entropy-based federated incremental learning and optimization in industrial internet of things[J]. Journal of Electronics & Information Technology, 2024, 46(8): 3146–3154. doi: 10.11999/JEIT231240.
    [7] MCMAHAN B, MOORE E, RAMAGE D, et al. Communication-efficient learning of deep networks from decentralized data[C]. Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, Fort Lauderdale, USA, 2017: 1273–1282.
    [8] 何帆, 王勇, 杨静, 等. 面向边缘计算场景的个性化联邦学习综述[J]. 通信学报, 2025, 46(7): 206–225. doi: 10.11959/j.issn.1000-436x.2025131.

    HE Fan, WANG Yong, YANG Jing, et al. Survey of personalized federated learning for edge computing scenarios[J]. Journal on Communications, 2025, 46(7): 206–225. doi: 10.11959/j.issn.1000-436x.2025131.
    [9] FALLAH A, MOKHTARI A, and OZDAGLAR A E. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach[C]. Proceedings of the 34th International Conference on Neural Information Processing Systems, Vancouver, Canada, 2020: 300.
    [10] LI Tian, HU Shengyuan, BEIRAMI A, et al. Ditto: Fair and robust federated learning through personalization[C]. The 38th International Conference on Machine Learning (ICML), 2021: 6357–6368. (查阅网上资料, 未找到对应的出版地信息, 请确认补充).
    [11] LI Xiaoxiao, JIANG Meirui, ZHANG Xiaofei, et al. FedBN: Federated learning on non-IID features via local batch normalization[C]. The 9th International Conference on Learning Representations (ICLR), Virtual Event, Austria, 2021. (查阅网上资料, 出版地信息不确定, 请确认).
    [12] COLLINS L, HASSANI H, MOKHTARI A, et al. Exploiting shared representations for personalized federated learning[C]. The 38th International Conference on Machine Learning (ICML), 2021: 2089–2099. (查阅网上资料, 未找到对应的出版地信息, 请确认补充).
    [13] LI Qinbin, HE Bingsheng, and SONG D. Model-contrastive federated learning[C]. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, USA, 2021: 10708–10717. doi: 10.1109/CVPR46437.2021.01057.
    [14] LI Daliang and WANG Junpu. FedMD: Heterogenous federated learning via model distillation[C]. Proceedings of the NeurIPS 2019 Workshop on Federated Learning for Data Privacy and Confidentiality, Vancouver, Canada, 2019. (查阅网上资料, 出版地信息不确定, 请确认).
    [15] VENKATESWARAN P, ISAHAGIAN V, MUTHUSAMY V, et al. FedGen: Generalizable federated learning for sequential data[C]. 2023 IEEE 16th International Conference on Cloud Computing (CLOUD), Chicago, USA, 2023: 308–318.
    [16] HA D, DAI A M, and LE Q V. HyperNetworks[C]. The 5th International Conference on Learning Representations (ICLR), Toulon, France, 2017.
    [17] SHAMSIAN A, NAVON A, FETAYA E, et al. Personalized federated learning using hypernetworks[C]. The 38th International Conference on Machine Learning (ICML), 2021: 9489–9502. (查阅网上资料, 未找到对应的出版地信息, 请确认补充).
    [18] MA Xiaosong, ZHANG Jie, GUO Song, et al. Layer-wised model aggregation for personalized federated learning[C]. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, USA, 2022: 10082–10091. doi: 10.1109/CVPR52688.2022.00985.
    [19] LI Hongxia, CAI Zhongyi, WANG Jingya, et al. FedTP: Federated learning by transformer personalization[J]. IEEE Transactions on Neural Networks and Learning Systems, 2024, 35(10): 13426–13440. doi: 10.1109/TNNLS.2023.3269062.
    [20] 李阳, 姜毅, 陈帅, 等. 多层超网络聚合的个性化联邦学习算法[J]. 计算机工程, 2026, 52(9): 133–142. doi: 10.19678/j.issn.1000-3428.0070679.

    LI Yang, JIANG Yi, CHEN Shuai, et al. Personalized federated learning algorithm based on multi-layer hypernetwork aggregation[J]. Computer Engineering, 2026, 52(9): 133–142. doi: 10.19678/j.issn.1000-3428.0070679.
    [21] YANG Ziyuan, XIA Wenjun, LU Zexin, et al. Hypernetwork-based physics-driven personalized federated learning for CT imaging[J]. IEEE Transactions on Neural Networks and Learning Systems, 2025, 36(2): 3136–3150. doi: 10.1109/TNNLS.2023.3338867.
    [22] CHEN Leiming, ZHANG Weishan, DONG Cihao, et al. FedDRL: Trustworthy federated learning model fusion method based on staged reinforcement learning[J]. Computing and Informatics, 2024, 43(1): 1–37. doi: 10.31577/cai_2024_1_1.
    [23] BANG J, WOO S, and LEE J. DRL-empowered joint batch size and weighted aggregation adjustment mechanism for federated learning on non-IID data[J]. ICT Express, 2024, 10(4): 863–870. doi: 10.1016/j.icte.2024.04.011.
    [24] LILLICRAP T P, HUNT J J, PRITZEL A, et al. Continuous control with deep reinforcement learning[C]. Proceedings of the 4th International Conference on Learning Representations (ICLR), San Juan, Puerto Rico, 2016.
    [25] BOTTOU L, CURTIS F E, and NOCEDAL J. Optimization methods for large-scale machine learning[J]. SIAM Review, 2018, 60(2): 223–311. doi: 10.1137/16M1080173.
    [26] VIRMAUX A and SCAMAN K. Lipschitz regularity of deep neural networks: Analysis and efficient estimation[C]. Proceedings of the 32nd International Conference on Neural Information Processing Systems, Montréal, Canada, 2018: 3839–3848.
    [27] FUJIMOTO S, VAN HOOF H, and MEGER D. Addressing function approximation error in actor-critic methods[C]. The 35th International Conference on Machine Learning (ICML), Stockholm, Sweden, 2018: 1582–1591.
    [28] LECUN Y, BOTTOU L, BENGIO Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11): 2278–2324. doi: 10.1109/5.726791.
    [29] KRIZHEVSKY A. Learning multiple layers of features from tiny images[R]. Toronto, Canada: University of Toronto, 2009.
    [30] ANGUITA D, GHIO A, ONETO L, et al. A public domain dataset for human activity recognition using smartphones[C]. The 21st European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN), Bruges, Belgium, 2013: 437–442.
    [31] CALDAS S, DUDDU S M K, WU P, et al. LEAF: A benchmark for federated settings[C]. Workshop on Federated Learning for Data Privacy and Confidentiality at NeurIPS 2019, Vancouver, Canada, 2019. (查阅网上资料, 未找到本条文献信息, 请确认).
    [32] LI Tian, SAHU A K, ZAHEER M, et al. Federated optimization in heterogeneous networks[C]. The 3rd Conference on Machine Learning and Systems (MLSys), Austin, USA, 2020: 429–450. (查阅网上资料, 页码信息不确定, 请确认).
    [33] WU Chuhan, WU Fangzhao, LYU Lingjuan, et al. Communication-efficient federated learning via knowledge distillation[J]. Nature Communications, 2022, 13: 2032. doi: 10.1038/s41467-022-29763-x.
    [34] ARIVAZHAGAN M G, AGGARWAL V, SINGH A K, et al. Federated learning with personalization layers[EB/OL]. https://arxiv.org/abs/1912.00818, 2019.
  • 加载中
图(7) / 表(5)
计量
  • 文章访问数:  17
  • HTML全文浏览量:  3
  • PDF下载量:  0
  • 被引次数: 0
出版历程
  • 修回日期:  2026-11-02
  • 录用日期:  2026-11-02
  • 网络出版日期:  2026-11-14

目录

    /

    返回文章
    返回