Entropy-based Federated Incremental Learning and Optimization in Industrial Internet of Things

YANG Ruizhe; XIE Xinru; TENG Yinglei; LI Meng; SUN Yanhua; ZHANG Dajun

doi:10.11999/JEIT231240

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2022 >

YANG Ruizhe, XIE Xinru, TENG Yinglei, LI Meng, SUN Yanhua, ZHANG Dajun. Entropy-based Federated Incremental Learning and Optimization in Industrial Internet of Things[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT231240

Citation:

YANG Ruizhe, XIE Xinru, TENG Yinglei, LI Meng, SUN Yanhua, ZHANG Dajun. Entropy-based Federated Incremental Learning and Optimization in Industrial Internet of Things[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT231240

Citation:

PDF( 4481 KB)

Entropy-based Federated Incremental Learning and Optimization in Industrial Internet of Things

doi: 10.11999/JEIT231240

YANG Ruizhe^{1, 2
,
,},
XIE Xinru^{1, 2},
TENG Yinglei³,
LI Meng^{1, 2},
SUN Yanhua^{1, 2},
ZHANG Dajun⁴

1.
Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
2.
Beijing Laboratory of Advanced Information Networks, Beijing University of Technology, Beijing 100124, China
3.
Electronic Engineering School, Beijing University of Posts and Telecommunications, Beijing 100083, China
4.
School of Information Technology, Carleton University, Ottawa K1S 5B6, Canada

Funds: The National Natural Science Foundation of China (62171062, 62371012)

Received Date: 2023-11-08
Rev Recd Date: 2024-04-21

Available Online: 2024-05-13

Abstract

Abstract

In the face of large-scale, diverse, and time-evolving data, as well as machine learning tasks in industrial production processes, a Federated Incremental Learning(FIL) and optimization method based on information entropy is proposed in this paper. Within the federated framework, local computing nodes utilize local data for model training, and compute the average entropy to be transmitted to the server to assist in identifying class-incremental tasks. The global server then selects local nodes for current round training based on the locally provided average entropy and makes decisions on task incrementality, followed by global model deployment and aggregation updates. The proposed method combines average entropy and thresholds for nodes selection in various situations, achieving stable model learning under low average entropy and incremental model expansion under high average entropy. Additionally, convex optimization is employed to adaptively adjust aggregation frequency and resource allocation in resource-constrained scenarios, ultimately achieving effective model convergence. Simulation results demonstrate that the proposed method accelerates model convergence and enhances training accuracy in different scenarios.
- Industrial Internet of Things (IIoT),
- Federated Incremental Learning (FIL),
- Information entropy

FullText(HTML)

References(24)

References

[1]	BJORNSON E and SANGUINETTI L. Scalable cell-free massive MIMO systems[J]. IEEE Transactions on Communications, 2020, 68(7): 4247–4261. doi: 10.1109/tcomm.2020.2987311.
[2]	YU Wanke and ZHAO Chunhui. Broad convolutional neural network based industrial process fault diagnosis with incremental learning capability[J]. IEEE Transactions on Industrial Electronics, 2020, 67(6): 5081–5091. doi: 10.1109/tie.2019.2931255.
[3]	HUO Ru, ZENG Shiqin, WANG Zhihao, et al. A comprehensive survey on Blockchain in industrial internet of things: Motivations, research progresses, and future challenges[J]. IEEE Communications Surveys & Tutorials, 2022, 24(1): 88–122. doi: 10.1109/comst.2022.3141490.
[4]	SU Hang, QI Wen, HU Yingbai, et al. An incremental learning framework for human-like redundancy optimization of anthropomorphic manipulators[J]. IEEE Transactions on Industrial Informatics, 2022, 18(3): 1864–1872. doi: 10.1109/tii.2020.3036693.
[5]	KIRKPATRICK J, PASCANU R, RABINOWITZ N, et al. Overcoming catastrophic forgetting in neural networks[J]. Proceedings of the National Academy of Sciences of the United States of America, 2017, 114(13): 3521–3526. doi: 10.1073/pnas.1611835114.
[6]	SHOHAM N, AVIDOR T, KEREN A, et al. Overcoming forgetting in federated learning on non-IID data[J]. arXiv: 1910.07796, 2019. doi: 10.48550/arXiv.1910.07796.
[7]	LI Zhizhong and HOIEM D. Learning without forgetting[C]. The 14th European Conference on Computer Vision, Amsterdam, The Netherlands, 2016: 614–629. doi: 10.1007/978-3-319-46493-0_37.
[8]	DONG Jiahua, WANG Lixu, FANG Zhen, et al. Federated class-incremental learning[C]. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, USA, 2022: 10154–10163. doi: 10.1109/cvpr52688.2022.00992.
[9]	MAZUR M, PUSTELNIK Ł, KNOP S, et al. Target layer regularization for continual learning using Cramer-Wold distance[J]. Information Sciences, 2022, 609: 1369–1380. doi: 10.1016/j.ins.2022.07.085.
[10]	REBUFFI S-A, KOLESNIKOV A, SPERL G, et al. iCaRL: Incremental classifier and representation learning[C]. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, USA, 2017: 5533–5542. doi: 10.1109/cvpr.2017.587.
[11]	WANG Qiang, LIU Jiayi, JI Zhong, et al. Hierarchical correlations replay for continual learning[J]. Knowledge-Based Systems, 2022, 250: 109052. doi: 10.1016/j.knosys.2022.109052.
[12]	JI Zhong, LI Jin, WANG Qiang, et al. Complementary calibration: Boosting general continual learning with collaborative distillation and self-supervision[J]. IEEE Transactions on Image Processing, 2023, 32: 657–667. doi: 10.1109/tip.2022.3230457.
[13]	HAO Meng, LI Hongwei, LUO Xizhao, et al. Efficient and privacy-enhanced federated learning for industrial artificial intelligence[J]. IEEE Transactions on Industrial Informatics, 2020, 16(10): 6532–6542. doi: 10.1109/tii.2019.2945367.
[14]	CHEN Zhixiong, YI Wenqiang, DENG Yansha, et al. Device scheduling for wireless federated learning with latency and representativity[C]. 2022 International Conference on Electrical, Computer, Communications and Mechatronics Engineering (ICECCME), Maldives, Maldives, 2022: 1–6. doi: 10.1109/iceccme55909.2022.9988590.
[15]	YANG Zhaohui, CHEN Mingzhe, SAAD W, et al. Energy efficient federated learning over wireless communication networks[J]. IEEE Transactions on Wireless Communications, 2021, 20(3): 1935–1949. doi: 10.1109/twc.2020.3037554.
[16]	DINH C T, TRAN N H, NGUYEN M N H, et al. Federated learning over wireless networks: Convergence analysis and resource allocation[J]. IEEE/ACM Transactions on Networking, 2021, 29(1): 398–409. doi: 10.1109/tnet.2020.3035770.
[17]	JING Shusen and XIAO Chengshan. Federated learning via over-the-air computation with statistical channel state information[J]. IEEE Transactions on Wireless Communications, 2022, 21(11): 9351–9365. doi: 10.1109/twc.2022.3175887.
[18]	YANG Ruizhe. The adaptive distributed learning based on homomorphic encryption and blockchain[J]. High Technology Letters, 2022, 28(4): 337–344. doi: 10.3772/j.issn.1006-6748.2022.04.001.
[19]	ZHAO Rui, SONG Jinming, YUAN Yufeng, et al. Maximum entropy population-based training for zero-shot human-AI coordination[C]. The AAAI Conference on Artificial Intelligence, Washington, USA, 2023: 6145–6153. doi: 10.1609/aaai.v37i5.25758.
[20]	ZHANG Peiying, WANG Chao, JIANG Chunxiao, et al. Deep reinforcement learning assisted federated learning algorithm for data management of IIoT[J]. IEEE Transactions on Industrial Informatics, 2021, 17(12): 8475–8484. doi: 10.1109/tii.2021.3064351.
[21]	WANG Shiqiang, TUOR T, SALONIDIS T, et al. Adaptive federated learning in resource constrained edge computing systems[J]. IEEE Journal on Selected Areas in Communications, 2019, 37(6): 1205–1221. doi: 10.1109/jsac.2019.2904348.
[22]	KRIZHEVSKY A and HINTON G. Learning multiple layers of features from tiny images[J]. Handbook of Systemic Autoimmune Diseases, 2009, 1(4).
[23]	SHAFIQ M and GU Zhaoquan. Deep residual learning for image recognition: A survey[J]. Applied Sciences, 2022, 12(18): 8972. doi: 10.3390/app12188972.
[24]	MINAEE S, KALCHBRENNER N, CAMBRIA E, et al. Deep learning-based text classification: A comprehensive review[J]. ACM Computing Surveys, 2022, 54(3): 62. doi: 10.1145/3439726.