Fourier Personalized Federated Learning Mechanism Based on Spectral Clustering
-
摘要: 为了缓解联邦学习中跨不同用户终端数据非独立同分布(non-IID)引起的负面影响，该文提出一种基于谱聚类的傅里叶个性化联邦学习算法。具体地，构建一个面向图像分类识别的云边端协同个性化联邦学习模型，提出在云端协同下通过谱聚类将用户终端划分为多个聚类域，以充分利用相似用户终端学到的知识提升模型性能。其次，设计边端协同的局部联邦学习方法，通过代理模型在用户终端对个性化局部模型执行恢复与再更新的操作，可有效恢复聚合过程中丢失的本地知识。进一步地，设计云边协同的傅里叶个性化联邦学习方法，即云服务器通过傅里叶变换将局部模型参数转换到频域空间上进行聚合，为每个边缘节点定制高质量的个性化局部模型，可使全局模型更适用于各个分布式用户终端。最后，实验结果表明，与现有相关算法相比，所提算法收敛速度更快，准确率提高了3%～13%。Abstract: To relieve the negative impacts caused by non-Independent and Identically Distributed (non-IID) data across different clients in federated learning, a spectral clustering-based Fourier personalized federated learning mechanism is proposed to overcome the performance drops from data heterogeneity. Specifically, a cloud-edge-end collaborative personalized federated learning model for image recognition is constructed, and in order to make full use of the knowledge learned by similar clients, the clients are divided into multiple clusters by spectral clustering under cloud-edge collaboration. Next, a local federated learning method based on edge-end collaboration is proposed, in which an agent model is used to perform the process of restoring and re-updating the personalized local model at the clients to restore the local knowledge loss during aggregation. Furthermore, a cloud-edge collaborative Fourier personalized federated learning method is proposed to adapt the global model to each distributed client. In this method, the cloud server converts the local model parameters to the frequency domain space for aggregation through Fourier transform, and customizes high-quality personalized local model for each edge node. Finally, the experimental results demonstrate that the proposed algorithm obtains competitive convergence speed compared with existing representative works and the accuracy is 3%～13% higher.
-
Key words:
- Edge computing /
- Federated learning /
- Spectral clustering /
- Fourier transform
-
算法1 基于谱聚类的傅里叶个性化联邦学习算法 输入：${D_i}(1 \le i \le N)$, ${D_p} $, $f $和通信轮数V. 输出：${f_i}(1 \le i \le N)$. 1. BEGIN 2. 用户终端i从云服务器下载初始训练模型$f $到本地; 3. 用户终端i利用本地私有数据${D_i} $训练更新本地模型${f_i} $; 4. 用户终端$i $将${D_p} $的预测结果矩阵${{\mathbf{P}}_i} = ({p_{i1}},{p_{i2}}, \cdots ,{p_{iR}}) $上传至
云服务器;5. 云服务器利用式(1)为用户终端计算相似矩阵${{\mathbf{S}}_{N \times N}} $; 6. 云服务器利用式(2)—式(8)，通过谱聚类将$N $个用户终端聚类
成$K $个互不相交的聚类域$\displaystyle\bigcup\limits_{k = 1}^K { {\text{cluste} }{ {\text{r} }_k} } = \left\{ {1,2, \cdots ,N} \right\}$;7. 设置迭代次数$v = 1 $; 8. 本轮迭代开始: 9. 用户终端$ i \in {\text{cluste}}{{\text{r}}_k} $将本地模型参数${f_i} $上传给同一边缘节点
${B_k} $，聚合为局部模型${\theta _k} $;10. 边缘节点将局部模型${\theta _k} $上传给云服务器; 11. 云服务器利用式 (13)—式(17) ，采用傅里叶聚合为每个边缘
节点${B_k} $定制相应的个性化局部模型${\theta _k} $;12. 用户终端$i $下载个性化局部模型${\theta _k} $到本地代理模型${d_i} $，利用
式(10)—式(12)恢复聚合过程中丢失的知识，将聚合到的知识
传输到本地模型；13. $v = v + 1 $； 14. 若$v < V $，重复步骤7—步骤13，否则跳出迭代; 15. 得到个性化本地模型参数${f_i}(1 \le i \le N)$. 16. END 
下载: 导出CSV
-
[1] MOHAMMADI M, AL-FUQAHA A, GUIZANI M, et al. Semisupervised deep reinforcement learning in support of IoT and smart city services[J]. IEEE Internet of Things Journal, 2018, 5(2): 624–635. doi: 10.1109/jiot.2017.2712560 [2] LI Chengxi, LI Gang, and VARSHNEY P K. Federated learning with soft clustering[J]. IEEE Internet of Things Journal, 2022, 9(10): 7773–7782. doi: 10.1109/JIOT.2021.3113927 [3] MCMAHAN B, MOORE E, RAMAGE D, et al. Communication-efficient learning of deep networks from decentralized data[C]. The 20th International Conference on Artificial Intelligence and Statistics, Fort Lauderdale, USA, 2017: 1273–1282. [4] FEKI I, AMMAR S, KESSENTINI Y, et al. Federated learning for COVID-19 screening from chest X-ray images[J]. Applied Soft Computing, 2021, 106: 107330. doi: 10.1016/j.asoc.2021.107330 [5] SILVA S, GUTMAN B A, ROMERO E, et al. Federated learning in distributed medical databases: Meta-analysis of large-scale subcortical brain data[C]. 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI), Venice, Italy, 2019: 270–274. [6] LIU Yang, HUANG Anbu, LUO Yun, et al. FedVision: An online visual object detection platform powered by federated learning[C]. The AAAI Conference on Artificial Intelligence, New York, USA, 2020: 13172–13179. [7] ZHAO Zhongyuan, FENG Chenyuan, HONG Wei, et al. Federated learning with non-IID data in wireless networks[J]. IEEE Transactions on Wireless Communications, 2022, 21(3): 1927–1942. doi: 10.1109/TWC.2021.3108197 [8] KULKARNI V, KULKARNI M, and PANT A. Survey of personalization techniques for federated learning[C]. The 4th World Conference on Smart Trends in Systems, Security and Sustainability (WorldS4), London, UK, 2020: 794–797. [9] YAN Zengqiang, WICAKSANA J, WANG Zhiwei, et al. Variation-aware federated learning with multi-source decentralized medical image data[J]. IEEE Journal of Biomedical and Health Informatics, 2021, 25(7): 2615–2628. doi: 10.1109/JBHI.2020.3040015 [10] HUANG Li, SHEA A L, QIAN Huining, et al. Patient clustering improves efficiency of federated machine learning to predict mortality and hospital stay time using distributed electronic medical records[J]. Journal of Biomedical Informatics, 2019, 99: 103291. doi: 10.1016/j.jbi.2019.103291 [11] SATTLER F, MÜLLER K R, and SAMEK W. Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints[J]. IEEE Transactions on Neural Networks and Learning Systems, 2021, 32(8): 3710–3722. doi: 10.1109/TNNLS.2020.3015958 [12] BRIGGS C, FAN Zhong, and ANDRAS P. Federated learning with hierarchical clustering of local updates to improve training on non-IID data[C]. 2020 International Joint Conference on Neural Networks (IJCNN), Glasgow, UK, 2020: 1–9. [13] EK S, PORTET F, LALANDA P, et al. Artifact: A federated learning aggregation algorithm for pervasive computing: Evaluation and comparison[C]. 2021 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops), Kassel, Germany, 2021: 448–449. [14] LI Xiaoxiao, JIANG Meirui, ZHANG Xiaofei, et al. FedBN: Federated learning on non-IID features via local batch normalization[C]. 9th International Conference on Learning Representations (ICLR), Vienna, Austria, 2021: 1–27. [15] CHEN Hongyou and CHAO Weilun. FedBE: Making Bayesian model ensemble applicable to federated learning[C]. 9th International Conference on Learning Representations (ICLR), Vienna, Austria, 2020: 1–21. [16] YE Dongdong, YU Rong, PAN Miao, et al. Federated learning in vehicular edge computing: A selective model aggregation approach[J]. IEEE Access, 2020, 8: 23920–23935. doi: 10.1109/ACCESS.2020.2968399 [17] HUANG Yutao, CHU Lingyang, ZHOU Zirui, et al. Personalized cross-silo federated learning on non-IID data[J/OL]. The AAAI Conference on Artificial Intelligence, 2021: 7865–7873. [18] ZHANG M, SAPRA K, FIDLER S, et al. Personalized federated learning with first order model optimization[C]. 9th International Conference on Learning Representations (ICLR), Vienna, Austria, 2020: 1–17. [19] CHEN Zhen, ZHU Meilu, YANG Chen, et al. Personalized retrogress-resilient framework for real-world medical federated learning[C]. The 24th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), Strasbourg, France, 2021: 347–356. -
下载:
点击查看大图
图(6) / 表(1)
计量
- 文章访问数: 390
- HTML全文浏览量: 257
- PDF下载量: 108
- 被引次数: 0
