Personalized Federated Learning Method Based on Collation Game and Knowledge Distillation

SUN Yanhua; SHI Yahui; LI Meng; YANG Ruizhe; SI Pengbo

doi:10.11999/JEIT221203

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2023 >

SUN Yanhua, SHI Yahui, LI Meng, YANG Ruizhe, SI Pengbo. Personalized Federated Learning Method Based on Collation Game and Knowledge Distillation[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT221203

Citation:

SUN Yanhua, SHI Yahui, LI Meng, YANG Ruizhe, SI Pengbo. Personalized Federated Learning Method Based on Collation Game and Knowledge Distillation[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT221203

Citation:

PDF( 1097 KB)

Personalized Federated Learning Method Based on Collation Game and Knowledge Distillation

doi: 10.11999/JEIT221203

SUN Yanhua^{1, 2
,
,},
SHI Yahui^{1, 2},
LI Meng^{1, 2},
YANG Ruizhe^{1, 2},
SI Pengbo^{1, 2}

1.
Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
2.
Beijing Laboratory of Advanced Information Networks, Beijing University of Technology, Beijing 100124, China

Funds: Foundation of Beijing Municipal Commission of Education (KM202010005017)

Received Date: 2022-09-20
Rev Recd Date: 2022-12-27

Available Online: 2022-12-28

Abstract

Abstract

To overcome the limitation of the Federated Learning (FL) when the data and model of each client are all heterogenous and improve the accuracy, a personalized Federated learning algorithm with Collation game and Knowledge distillation (pFedCK) is proposed. Firstly, each client uploads its soft-predict on public dataset and download the most correlative of the k soft-predict. Then, this method apply the shapley value from collation game to measure the multi-wise influences among clients and quantify their marginal contribution to others on personalized learning performance. Lastly, each client identify it’s optimal coalition and then distill the knowledge to local model and train on private dataset. The results show that compared with the state-of-the-art algorithm, this approach can achieve superior personalized accuracy and can improve by about 10%.
- Personalized Federated Learning(PFL),
- Collation game,
- Knowledge distillation,
- Heterogeneity

FullText(HTML)

References(15)

References

[1]	MCMAHAN B, MOORE E, RAMAGE D, et al. Communication-efficient learning of deep networks from decentralized data[C]. Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, Fort Lauderdale, USA, 2017: 1273–1282.
[2]	LI Tian, SAHU A K, TALWALKAR A, et al. Federated learning: Challenges, methods, and future directions[J]. IEEE Signal Processing Magazine, 2020, 37(3): 50–60. doi: 10.1109/MSP.2020.2975749
[3]	ARIVAZHAGAN M G, AGGARWAL V, SINGH A K, et al. Federated learning with personalization layers[EB/OL]. https://doi.org/10.48550/arXiv.1912.00818, 2019.
[4]	HANZELY F and RICHTÁRIK P. Federated learning of a mixture of global and local models[EB/OL]. https://doi.org/10.48550/arXiv.2002.05516, 2020.
[5]	FALLAH A, MOKHTARI A, and OZDAGLAR A E. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach[C]. Proceedings of the 34th International Conference on Neural Information Processing Systems, Vancouver, Canada, 2020: 3557–3568.
[6]	GHOSH A, CHUNG J, YIN D, et al. An efficient framework for clustered federated learning[C]. Proceedings of the 34th International Conference on Neural Information Processing Systems, Vancouver, Canada, 2020: 19586–19597.
[7]	WU Leijie, GUO Song, DING Yaohong, et al. A coalition formation game approach for personalized federated learning[EB/OL]. https://doi.org/10.48550/arXiv.2202.02502, 2022.
[8]	HINTON G, VINYALS O, and DEAN J. Distilling the knowledge in a neural network[J]. Computer Science, 2015, 14(7): 38–39.
[9]	LI Daliang and WANG Junpu. FedMD: Heterogenous federated learning via model distillation[EB/OL]. https://doi.org/10.48550/arXiv.1910.03581, 2019.
[10]	LIN Tao, KONG Lingjing, STICH S U, et al. Ensemble distillation for robust model fusion in federated learning[C]. Proceedings of the 34th International Conference on Neural Information Processing Systems, Vancouver, Canada, 2020: 2351–2363.
[11]	ZHANG Jie, GUO Song, MA Xiaosong, et al. Parameterized knowledge transfer for personalized federated learning[C]. Proceedings of the 35th International Conference on Neural Information Processing Systems, Vancouver, Canada, 2021: 10092–10104.
[12]	CHO Y J, WANG Jianyu, CHIRUVOLU T, et al. Personalized federated learning for heterogeneous clients with clustered knowledge transfer[EB/OL]. https://doi.org/10.48550/arXiv.2109.08119, 2021.
[13]	DONAHUE K and KLEINBERG J. Model-sharing games: Analyzing federated learning under voluntary participation[C]. Proceedings of the 35th AAAI Conference on Artificial Intelligence, Vancouver, Canada, 2021: 5303–5311.
[14]	LUNDBERG S M and LEE S I. A unified approach to interpreting model predictions[C]. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, USA, 2017: 4768–4777.
[15]	SAAD W, HAN Zhu, DEBBAH M, et al. Coalitional game theory for communication networks[J]. IEEE Signal Processing Magazine, 2009, 26(5): 77–97. doi: 10.1109/MSP.2009.000000