异构物联网下资源高效的分层协同联邦学习方法

王汝言; 陈伟; 张普宁; 吴大鹏; 杨志刚

doi:10.11999/JEIT220914

异构物联网下资源高效的分层协同联邦学习方法

doi: 10.11999/JEIT220914

1.
重庆邮电大学通信与信息工程学院重庆 400065
2.
先进网络与智能互联技术重庆市高校重点实验室重庆 400065
3.
泛在感知与互联重庆市重点实验室重庆 400065

基金项目: 国家自然科学基金(61901071, 61871062, 61771082, U20A20157)，重庆市自然科学基金(cstc2020jcyj-zdxmX0024)，重庆市高校创新研究群体(CXQT20017)，重庆高校创新团队建设计划(CXTDX201601020)

详细信息

作者简介:
王汝言：男，教授，研究方向为泛在网络、多媒体信息处理等

陈伟：男，硕士生，研究方向为联邦学习

张普宁：男，副教授，研究方向为物联网搜索等

吴大鹏：男，教授，研究方向为泛在无线网络、社会计算等

杨志刚：男，博士，研究方向为隐私计算等

通讯作者:
张普宁　zhangpn@cqupt.edu.cn

中图分类号: TN915; TP399
计量
- 文章访问数: 1053
- HTML全文浏览量: 737
- PDF下载量: 257
- 被引次数: 0
出版历程
- 收稿日期: 2022-07-06
- 修回日期: 2022-08-21
- 网络出版日期: 2022-09-02
- 刊出日期: 2023-08-21

Resource-Efficient Hierarchical Collaborative Federated Learning in Heterogeneous Internet of Things

1.
School of Communication and Information Engineering, Chongqing University of Posts and Telecommunications, Chongqing 400065, China
2.
Advanced Network and Intelligent Connection Technology Key Laboratory of Chongqing Education Commission of China, Chongqing 400065, China
3.
Chongqing Key Laboratory of Ubiquitous Sensing and Networking, Chongqing 400065, China

Funds: The National Natural Science Foundation of China (61901071, 61871062, 61771082, U20A20157), The Science and Natural Science Foundation of Chongqing (cstc2020jcyj-zdxmX0024), The University Innovation Research Group of Chongqing (CXQT20017), The Program for Innovation Team Building at Institutions of Higher Education in Chongqing (CXTDX201601020)

摘要

摘要: 物联网(IoT)设备资源存在高度异构性，严重影响联邦学习(FL)的训练时间和精度。已有研究未充分考虑物联网设备资源的异构性，且缺乏异构设备间协同训练机制的设计，导致训练效果有限且设备的资源利用率较低。为此，该文提出资源高效的分层协同联邦学习方法(HCFL)，设计了端边云分层混合聚合机制，考虑边缘服务器的差异化参数聚合频率，提出自适应异步加权聚合方法，提高模型参数聚合效率。提出资源重均衡的客户端选择算法，考虑模型精度与数据分布特征动态选取客户端，缓解资源异构性对联邦学习性能的影响。设计自组织联邦协同训练算法，充分利用空闲物联网设备资源加速联邦学习训练进程。仿真结果表明，在不同资源异构状态下，与基线方法相比，模型训练时间平均降低15％，模型精度平均提高6％，设备平均资源利用率提高52％。
- 异构物联网 /
- 边缘智能 /
- 分层联邦学习 /
- 客户端选择
Abstract: The high heterogeneity of Internet of Things (IoT) devices and resources affects severely the training efficiency and accuracy of Federated Learning (FL). This characteristical heterogeneity of IoT devices and resources is fully investigated by existing research, and the design of a collaborative training acceleration mechanism among heterogeneous IoT devices is rare, resulting in limited training efficiency and low resource utilization of IoT devices. To this end, a resource-efficient Hierarchical Collaborative Federated Learning (HCFL) approach is proposed, and a device-edge-cloud hierarchical hybrid aggregation mechanism is devised, including an adaptive asynchronous weighted aggregation method to improve the model parameter aggregation efficiency by exploiting the differentiated parameter aggregation frequency of edge servers. A resource rebalancing client selection algorithm is proposed to select dynamically clients considering model accuracy and data distribution characteristics to mitigate the impact of resource heterogeneity on FL performance. A self-organized collaborative training algorithm is designed to leverage idle IoT devices and resources to accelerate the FL training process. Simulation results show that, given different heterogeneity degrees, the average training time of FL models is reduced by 15%, the average accuracy of FL models is improved by 6%, and the average resource utilization of IoT devices is improved by 52%.
- Heterogeneous Internet of Things (IoT) /
- Edge artificial intelligence /
- Hierarchical federated learning /
- Client selection

HTML全文

图 1 联邦学习系统架构图

下载: 全尺寸图片幻灯片

图 2 模型协作训练流程

下载: 全尺寸图片幻灯片

图 3 Mnist资源重均衡客户端选择算法对比

下载: 全尺寸图片幻灯片

图 4 Fashion Mnist资源重均衡客户端选择算法对比

下载: 全尺寸图片幻灯片

图 5 不同设备异构情况下协作集合发现

下载: 全尺寸图片幻灯片

图 6 不同协作策略训练时间对比

下载: 全尺寸图片幻灯片

图 7 分层混合聚合机制对比

下载: 全尺寸图片幻灯片

参考文献(23)

[1]	ZHANG Jing and TAO Dacheng. Empowering things with intelligence: a survey of the progress, challenges, and opportunities in artificial intelligence of things[J]. IEEE Internet of Things Journal, 2021, 8(10): 7789–7817. doi: 10.1109/JIOT.2020.3039359
[2]	WANG Xiaofei, HAN Yiwen, LEUNG V C M, et al. Convergence of edge computing and deep learning: A comprehensive survey[J]. IEEE Communications Surveys & Tutorials, 2020, 22(2): 869–904. doi: 10.1109/COMST.2020.2970550
[3]	YANG Qiang, LIU Yang, CHEN Tianjian, et al. Federated machine learning: Concept and applications[J]. ACM Transactions on Intelligent Systems and Technology, 2019, 10(2): 12. doi: 10.1145/3298981
[4]	MCMAHAN B, MOORE E, RAMAGE D, et al. Communication-efficient learning of deep networks from decentralized data[C]. Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, Fort Lauderdale, USA, 2017: 1273–1282.
[5]	LIU Lumin, ZHANG Jun, SONG S H, et al. Client-edge-cloud hierarchical federated learning[C]. Proceedings of 2020 IEEE International Conference on Communications (ICC), Dublin, Ireland, 2020: 1–6.
[6]	CHAI Zheng, ALI A, ZAWAD S, et al. TiFL: A tier-based federated learning system[C]. Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing, Stockholm, Sweden, 2020: 125–136.
[7]	ZAWAD S, ALI A, CHEN Pinyu, et al. Curse or redemption? How data heterogeneity affects the robustness of federated learning[C]. Proceedings of the 35th AAAI Conference on Artificial Intelligence, Vancouver, Canada, 2021: 10807–10814.
[8]	DUAN Moming, LIU Duo, CHEN Xianzhang, et al. Self-balancing federated learning with global imbalanced data in mobile systems[J]. IEEE Transactions on Parallel and Distributed Systems, 2021, 32(1): 59–71. doi: 10.1109/TPDS.2020.3009406
[9]	NISHIO T and YONETANI R. Client selection for federated learning with heterogeneous resources in mobile edge[C]. Proceedings of 2019 IEEE International Conference on Communications (ICC), Shanghai, China, 2019: 1–7.
[10]	PARK J, HAN D J, CHOI M, et al. Sageflow: Robust federated learning against both stragglers and adversaries[C/OL]. Proceedings of the 35th Conference on Neural Information Processing Systems, 2021: 840–851.
[11]	DINH C T, TRAN N H, NGUYEN M N H, et al. Federated learning over wireless networks: Convergence analysis and resource allocation[J]. IEEE/ACM Transactions on Networking, 2021, 29(1): 398–409. doi: 10.1109/TNET.2020.3035770
[12]	ABAD M S H, OZFATURA E, GUNDUZ D, et al. Hierarchical federated learning ACROSS heterogeneous cellular networks[C]. Proceedings of 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain, 2020: 8866–8870.
[13]	ZHAO Yue, LI Meng, LAI Liangzhen, et al. Federated learning with non-IID data[EB/OL]. https://doi.org/10.48550/arXiv.1806.00582, 2018.
[14]	IMTEAJ A, THAKKER U, WANG Shiqiang, et al. A survey on federated learning for resource-constrained IoT devices[J]. IEEE Internet of Things Journal, 2022, 9(1): 1–24. doi: 10.1109/JIOT.2021.3095077
[15]	LI Ang, SUN Jingwei, LI Pengcheng, et al. Hermes: An efficient federated learning framework for heterogeneous mobile clients[C]. Proceedings of the 27th Annual International Conference on Mobile Computing and Networking, New Orleans Louisiana, USA, 2021: 420–437.
[16]	WU Di, ULLAH R, HARVEY P, et al. FedAdapt: Adaptive offloading for IoT devices in federated learning[J]. IEEE Internet of Things Journal, 2022, 9(21): 20889–20901. doi: 10.1109/JIOT.2022.3176469
[17]	HUANG Yakun, QIAO Xiuquan, DUSTDAR S, et al. Toward decentralized and collaborative deep learning inference for intelligent IoT devices[J]. IEEE Network, 2022, 36(1): 59–68. doi: 10.1109/MNET.011.2000639
[18]	ZENG Liekang, CHEN Xu, ZHOU Zhi, et al. CoEdge: Cooperative DNN inference with adaptive workload partitioning over heterogeneous edge devices[J]. IEEE/ACM Transactions on Networking, 2021, 29(2): 595–608. doi: 10.1109/TNET.2020.3042320
[19]	SCHULMAN J, WOLSKI F, DHARIWAL P, et al. Proximal policy optimization algorithms[J]. arXiv: 1707.06347, 2017.
[20]	HE Chaoyang, LI Songze, SO J, et al. FedML: A research library and benchmark for federated machine learning[J]. arXiv: 2007.13518, 2020.
[21]	LECUN Y, BOTTOU L, BENGIO Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11): 2278–2324. doi: 10.1109/5.726791
[22]	XIAO Han, RASUL K, and VOLLGRAF R. Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms[EB/OL]. https://doi.org/10.48550/arXiv.1708.07747, 2017.
[23]	YOSHIDA N, NISHIO T, MORIKURA M, et al. Hybrid-FL for wireless networks: Cooperative learning mechanism using non-IID data[C]. Proceedings of 2020 IEEE International Conference on Communications (ICC), Dublin, Ireland, 2020: 1–7.