Resource-Efficient Hierarchical Collaborative Federated Learning in Heterogeneous Internet of Things

WANG Ruyan; CHEN Wei; ZHANG Puning; WU Dapeng; YANG Zhigang

doi:10.11999/JEIT220914

Volume 45 Issue 8

Aug. 2023

Turn off MathJax

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2023 > 45(8): 2847-2855

WANG Ruyan, CHEN Wei, ZHANG Puning, WU Dapeng, YANG Zhigang. Resource-Efficient Hierarchical Collaborative Federated Learning in Heterogeneous Internet of Things[J]. Journal of Electronics & Information Technology, 2023, 45(8): 2847-2855. doi: 10.11999/JEIT220914

Citation:

WANG Ruyan, CHEN Wei, ZHANG Puning, WU Dapeng, YANG Zhigang. Resource-Efficient Hierarchical Collaborative Federated Learning in Heterogeneous Internet of Things[J]. Journal of Electronics & Information Technology, 2023, 45(8): 2847-2855. doi: 10.11999/JEIT220914

Citation:

PDF( 6032 KB)

Resource-Efficient Hierarchical Collaborative Federated Learning in Heterogeneous Internet of Things

doi: 10.11999/JEIT220914

1.
School of Communication and Information Engineering, Chongqing University of Posts and Telecommunications, Chongqing 400065, China
2.
Advanced Network and Intelligent Connection Technology Key Laboratory of Chongqing Education Commission of China, Chongqing 400065, China
3.
Chongqing Key Laboratory of Ubiquitous Sensing and Networking, Chongqing 400065, China

Funds: The National Natural Science Foundation of China (61901071, 61871062, 61771082, U20A20157), The Science and Natural Science Foundation of Chongqing (cstc2020jcyj-zdxmX0024), The University Innovation Research Group of Chongqing (CXQT20017), The Program for Innovation Team Building at Institutions of Higher Education in Chongqing (CXTDX201601020)

Received Date: 2022-07-06
Rev Recd Date: 2022-08-21

Available Online: 2022-09-02

Publish Date: 2023-08-21

Abstract

Abstract

The high heterogeneity of Internet of Things (IoT) devices and resources affects severely the training efficiency and accuracy of Federated Learning (FL). This characteristical heterogeneity of IoT devices and resources is fully investigated by existing research, and the design of a collaborative training acceleration mechanism among heterogeneous IoT devices is rare, resulting in limited training efficiency and low resource utilization of IoT devices. To this end, a resource-efficient Hierarchical Collaborative Federated Learning (HCFL) approach is proposed, and a device-edge-cloud hierarchical hybrid aggregation mechanism is devised, including an adaptive asynchronous weighted aggregation method to improve the model parameter aggregation efficiency by exploiting the differentiated parameter aggregation frequency of edge servers. A resource rebalancing client selection algorithm is proposed to select dynamically clients considering model accuracy and data distribution characteristics to mitigate the impact of resource heterogeneity on FL performance. A self-organized collaborative training algorithm is designed to leverage idle IoT devices and resources to accelerate the FL training process. Simulation results show that, given different heterogeneity degrees, the average training time of FL models is reduced by 15%, the average accuracy of FL models is improved by 6%, and the average resource utilization of IoT devices is improved by 52%.
- Heterogeneous Internet of Things (IoT),
- Edge artificial intelligence,
- Hierarchical federated learning,
- Client selection

FullText(HTML)

References(23)

References

[1]	ZHANG Jing and TAO Dacheng. Empowering things with intelligence: a survey of the progress, challenges, and opportunities in artificial intelligence of things[J]. IEEE Internet of Things Journal, 2021, 8(10): 7789–7817. doi: 10.1109/JIOT.2020.3039359
[2]	WANG Xiaofei, HAN Yiwen, LEUNG V C M, et al. Convergence of edge computing and deep learning: A comprehensive survey[J]. IEEE Communications Surveys & Tutorials, 2020, 22(2): 869–904. doi: 10.1109/COMST.2020.2970550
[3]	YANG Qiang, LIU Yang, CHEN Tianjian, et al. Federated machine learning: Concept and applications[J]. ACM Transactions on Intelligent Systems and Technology, 2019, 10(2): 12. doi: 10.1145/3298981
[4]	MCMAHAN B, MOORE E, RAMAGE D, et al. Communication-efficient learning of deep networks from decentralized data[C]. Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, Fort Lauderdale, USA, 2017: 1273–1282.
[5]	LIU Lumin, ZHANG Jun, SONG S H, et al. Client-edge-cloud hierarchical federated learning[C]. Proceedings of 2020 IEEE International Conference on Communications (ICC), Dublin, Ireland, 2020: 1–6.
[6]	CHAI Zheng, ALI A, ZAWAD S, et al. TiFL: A tier-based federated learning system[C]. Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing, Stockholm, Sweden, 2020: 125–136.
[7]	ZAWAD S, ALI A, CHEN Pinyu, et al. Curse or redemption? How data heterogeneity affects the robustness of federated learning[C]. Proceedings of the 35th AAAI Conference on Artificial Intelligence, Vancouver, Canada, 2021: 10807–10814.
[8]	DUAN Moming, LIU Duo, CHEN Xianzhang, et al. Self-balancing federated learning with global imbalanced data in mobile systems[J]. IEEE Transactions on Parallel and Distributed Systems, 2021, 32(1): 59–71. doi: 10.1109/TPDS.2020.3009406
[9]	NISHIO T and YONETANI R. Client selection for federated learning with heterogeneous resources in mobile edge[C]. Proceedings of 2019 IEEE International Conference on Communications (ICC), Shanghai, China, 2019: 1–7.
[10]	PARK J, HAN D J, CHOI M, et al. Sageflow: Robust federated learning against both stragglers and adversaries[C/OL]. Proceedings of the 35th Conference on Neural Information Processing Systems, 2021: 840–851.
[11]	DINH C T, TRAN N H, NGUYEN M N H, et al. Federated learning over wireless networks: Convergence analysis and resource allocation[J]. IEEE/ACM Transactions on Networking, 2021, 29(1): 398–409. doi: 10.1109/TNET.2020.3035770
[12]	ABAD M S H, OZFATURA E, GUNDUZ D, et al. Hierarchical federated learning ACROSS heterogeneous cellular networks[C]. Proceedings of 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain, 2020: 8866–8870.
[13]	ZHAO Yue, LI Meng, LAI Liangzhen, et al. Federated learning with non-IID data[EB/OL]. https://doi.org/10.48550/arXiv.1806.00582, 2018.
[14]	IMTEAJ A, THAKKER U, WANG Shiqiang, et al. A survey on federated learning for resource-constrained IoT devices[J]. IEEE Internet of Things Journal, 2022, 9(1): 1–24. doi: 10.1109/JIOT.2021.3095077
[15]	LI Ang, SUN Jingwei, LI Pengcheng, et al. Hermes: An efficient federated learning framework for heterogeneous mobile clients[C]. Proceedings of the 27th Annual International Conference on Mobile Computing and Networking, New Orleans Louisiana, USA, 2021: 420–437.
[16]	WU Di, ULLAH R, HARVEY P, et al. FedAdapt: Adaptive offloading for IoT devices in federated learning[J]. IEEE Internet of Things Journal, 2022, 9(21): 20889–20901. doi: 10.1109/JIOT.2022.3176469
[17]	HUANG Yakun, QIAO Xiuquan, DUSTDAR S, et al. Toward decentralized and collaborative deep learning inference for intelligent IoT devices[J]. IEEE Network, 2022, 36(1): 59–68. doi: 10.1109/MNET.011.2000639
[18]	ZENG Liekang, CHEN Xu, ZHOU Zhi, et al. CoEdge: Cooperative DNN inference with adaptive workload partitioning over heterogeneous edge devices[J]. IEEE/ACM Transactions on Networking, 2021, 29(2): 595–608. doi: 10.1109/TNET.2020.3042320
[19]	SCHULMAN J, WOLSKI F, DHARIWAL P, et al. Proximal policy optimization algorithms[J]. arXiv: 1707.06347, 2017.
[20]	HE Chaoyang, LI Songze, SO J, et al. FedML: A research library and benchmark for federated machine learning[J]. arXiv: 2007.13518, 2020.
[21]	LECUN Y, BOTTOU L, BENGIO Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11): 2278–2324. doi: 10.1109/5.726791
[22]	XIAO Han, RASUL K, and VOLLGRAF R. Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms[EB/OL]. https://doi.org/10.48550/arXiv.1708.07747, 2017.
[23]	YOSHIDA N, NISHIO T, MORIKURA M, et al. Hybrid-FL for wireless networks: Cooperative learning mechanism using non-IID data[C]. Proceedings of 2020 IEEE International Conference on Communications (ICC), Dublin, Ireland, 2020: 1–7.