A knowledge distillation framework for hypergraph neural networks with rapid inference capabilities

LI Junzheng; YU Hongtao; HUANG Ruiyang; JIANG Haocong; LIU Shuo; YANG Suchang

doi:10.11999/JEIT260694

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2026 >

LI Junzheng, YU Hongtao, HUANG Ruiyang, JIANG Haocong, LIU Shuo, YANG Suchang. A knowledge distillation framework for hypergraph neural networks with rapid inference capabilities[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260694

Citation:

LI Junzheng, YU Hongtao, HUANG Ruiyang, JIANG Haocong, LIU Shuo, YANG Suchang. A knowledge distillation framework for hypergraph neural networks with rapid inference capabilities[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260694

Citation:

LI Junzheng, YU Hongtao, HUANG Ruiyang, JIANG Haocong, LIU Shuo, YANG Suchang. A knowledge distillation framework for hypergraph neural networks with rapid inference capabilities[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260694

PDF( 1485 KB)

A knowledge distillation framework for hypergraph neural networks with rapid inference capabilities

doi: 10.11999/JEIT260694 cstr: 32379.14.JEIT260694

Information Engineering University, Zhengzhou 450000, China

Accepted Date: 2026-08-24
Rev Recd Date: 2026-08-24

Available Online: 2026-09-01

Abstract

Abstract

Objective Hypergraph Neural Networks (HGNNs) have gained widespread attention for their strong ability to model high-order correlations among entities, but their computational complexity and memory consumption grow exponentially as the hypergraph scale expands, severely restricting their deployment in large-scale industrial scenarios. Existing knowledge distillation methods that distill HGNNs into Multi-Layer Perceptrons (MLPs) are troubled by poor interpretability, low accuracy, severe information loss caused by Softmax-based soft labels, and neglect of node reliability heterogeneity. To address these critical challenges, this paper proposes a novel hypergraph knowledge distillation framework named DH2KAN (Distill Hypergraph Neural Network to Kolmogorov-Arnold Network) for fast inference, which breaks through the bottlenecks of traditional hypergraph knowledge distillation. Methods This paper designs a three-module knowledge distillation framework DH2KAN(图2). Firstly, we replace the traditional MLP with KAN as the student model, which uses learnable spline-based univariate functions instead of fixed activation functions and linear weights to improve the fitting ability and interpretability of the student model. Secondly, we propose a representation similarity distillation mechanism, which directly aligns the pre-logits representations of HGNN and KAN to avoid information loss caused by the Softmax normalization layer and completely retain the high-order structural knowledge of hypergraphs. Thirdly, we introduce a high-reliable node-aware distillation method(图3), which quantifies the node reliability by information entropy variation, screens out robust nodes with strong anti-noise ability, and takes their soft labels as the core supervision signal to improve the purity of distilled knowledge. Results and Discussions The DH2KAN algorithm achieves remarkable performance under both transductive learning (表2) and production learning (表3) settings. Quantitative experimental results reveal that DH2KAN obtains an accuracy improvement of approximately 10.53% over the vanilla KAN student model, 1.73% over the teacher HGNN model, and 1.2% over existing MLP-based distillation methods. Such results verify the effectiveness of knowledge transfer from HGNN to KAN, and demonstrate that the proposed method outperforms conventional MLP-oriented distillation schemes in inference performance. In addition, DH2KAN achieves optimal performance on feature-dominated hypergraph datasets, and possesses strong robustness when handling sparse structures and noisy node samples. Conclusions This paper proposes DH2KAN to accelerate HGNN inference for large-scale low-latency applications. Via knowledge distillation, it bridges the performance gap between KAN and HGNN and eliminates structural dependence for efficient reasoning. With representation similarity and reliable node-aware distillation, it transfers effective task knowledge via pre-logit features and soft labels, showing great practical application potential.
- Hypergraph Neural Network,
- Kolmogorov-Arnold Network,
- Knowledge Distillation,
- Fast Inference

FullText(HTML)

References(31)

References

[1]	谢丽霞, 史镜琛, 杨宏宇, 等. 基于图神经网络模型校准的成员推理攻击[J]. 电子与信息学报, 2025, 47(3): 780–791. doi: 10.11999/JEIT240477. XIE Lixia, SHI Jingchen, YANG Hongyu, et al. Membership inference attacks based on graph neural network model calibration[J]. Journal of Electronics & Information Technology, 2025, 47(3): 780–791. doi: 10.11999/JEIT240477.
[2]	吴翼腾, 刘伟, 于洪涛, 等. 基于局部影响分析模型的图神经网络对抗攻击[J]. 电子与信息学报, 2022, 44(7): 2576–2583. doi: 10.11999/JEIT210448. WU Yiteng, LIU Wei, YU Hongtao, et al. Adversarial attacks on graph neural network based on local influence analysis model[J]. Journal of Electronics & Information Technology, 2022, 44(7): 2576–2583. doi: 10.11999/JEIT210448.
[3]	GAO Yue, ZHANG Zizhao, LIN Haojie, et al. Hypergraph learning: Methods and practices[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(5): 2548–2566. doi: 10.1109/TPAMI.2020.3039374.
[4]	TIAN Yijun, PEI Shichao, ZHANG Xiangliang, et al. Knowledge distillation on graphs: A survey[J]. ACM Computing Surveys, 2025, 57(8): 189. doi: 10.1145/3711121.
[5]	FENG Yifan, LUO Yihe, YING Shihui, et al. LightHGNN: Distilling hypergraph neural networks into MLPs for 100x faster inference[C]. The Twelfth International Conference on Learning Representations, Vienna, Austria, 2024.
[6]	LIU Ziming, WANG Yixuan, VAIDYA S, et al. KAN: Kolmogorov–Arnold networks[C]. The Thirteenth International Conference on Learning Representations, Singapore, Singapore, 2025: 70367–70413.
[7]	LIAO Sihao, XIE Liang, DU Yuanchuang, et al. Stock trend prediction based on dynamic hypergraph spatio-temporal network[J]. Applied Soft Computing, 2024, 154: 111329. doi: 10.1016/J.ASOC.2024.111329.
[8]	KIM S, LEE S Y, GAO Yue, et al. A survey on hypergraph neural networks: An in-depth and step-by-step guide[C]. Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Barcelona, Spain, 2024: 6534–6544.
[9]	BERGE C. Hypergraphs: Combinatorics of Finite Sets[M]. Amsterdam: Elsevier, 1989. (查阅网上资料, 请补充引用页码并核对出版年信息).
[10]	FENG Yifan, YOU Haoxuan, ZHANG Zizhao, et al. Hypergraph neural networks[C]. Proceedings of the 33rd AAAI Conference on Artificial Intelligence, Honolulu, USA, 2019: 3558–3565.
[11]	GAO Yue, FENG Yifan, JI Shuyi, et al. HGNN⁺: General hypergraph neural networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(3): 3181–3199. doi: 10.1109/TPAMI.2022.3182052.
[12]	YADATI N, NIMISHAKAVI M, YADAV P, et al. HyperGCN: A new method of training graph convolutional networks on hypergraphs[C]. Proceedings of the 33rd International Conference on Neural Information Processing Systems, Vancouver, Canada, 2019: 135.
[13]	DONG Yihe, SAWIN W, and BENGIO Y. HNHN: Hypergraph networks with hyperedge neurons[J]. arXiv: 2006.12278, 2020. doi: 10.48550/arXiv.2006.12278.
[14]	JIANG Jianwen, WEI Yuxuan, FENG Yifan, et al. Dynamic hypergraph neural networks[C]. Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, Macao, China, 2019: 2635–2641. doi: 10.24963/ijcai.2019/366.
[15]	BAI Song, ZHANG Feihu, and TORR P H S. Hypergraph convolution and hypergraph attention[J]. Pattern Recognition, 2021, 110: 107637. doi: 10.1016/j.patcog.2020.107637.
[16]	JO J, BAEK J, LEE S, et al. Edge representation learning with hypergraphs[C]. Proceedings of the 35th International Conference on Neural Information Processing Systems, 2021: 577. (查阅网上资料, 未找到本条文献出版地信息, 请确认并补充).
[17]	HUANG Jing and YANG Jie. UniGNN: A unified framework for graph and hypergraph neural networks[C]. Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, Montreal, Canada, 2021: 2563–2569. doi: 10.24963/IJCAI.2021/353.
[18]	CHIEN E, PAN Chao, PENG Jianhao, et al. You are AllSet: A multiset function framework for hypergraph neural networks[C]. The Tenth International Conference on Learning Representations, 2022. (查阅网上资料, 未找到本条文献出版地信息, 请确认并补充).
[19]	HINTON G, VINYALS O, and DEAN J. Distilling the knowledge in a neural network[J]. arXiv: 1503.02531, 2015. doi: 10.48550/arXiv.1503.02531.
[20]	罗一畅, 齐析屿, 张博锐, 等. 分割一切模型的轻量化研究综述[J]. 电子与信息学报, 2026, 48(2): 713–731. doi: 10.11999/JEIT250894. LUO Yichang, QI Xiyu, ZHANG Borui, et al. A survey of lightweight techniques for segment anything model[J]. Journal of Electronics & Information Technology, 2026, 48(2): 713–731. doi: 10.11999/JEIT250894.
[21]	ZHANG Shichang, LIU Yozen, SUN Yizhou, et al. Graph-less neural networks: Teaching old MLPs new tricks via distillation[C]. The Tenth International Conference on Learning Representations, 2022. (查阅网上资料, 未找到本条文献出版地信息, 请确认并补充).
[22]	YANG Cheng, LIU Jiawei, and SHI Chuan. Extract the knowledge of graph neural networks and go beyond it: An effective knowledge distillation framework[C]. Proceedings of the Web Conference 2021, Ljubljana, Slovenia, 2021: 1227–1237. doi: 10.1145/3442381.3450068.
[23]	WU Lirong, LIN Haitao, HUANG Yufei, et al. Quantifying the knowledge in GNNs for reliable distillation into MLPs[C]. Proceedings of the 40th International Conference on Machine Learning, Honolulu, USA, 2023: 37571–37581.
[24]	LIU Ziming, MA Pingchuan, WANG Yixuan, et al. KAN 2.0: Kolmogorov-Arnold networks meet science[J]. arXiv: 2408.10205, 2024. doi: 10.48550/arxiv.2408.10205.
[25]	郑庆河, 刘方霖, 余礼苏, 等. 基于改进Kolmogorov-Arnold混合卷积神经网络的调制识别方法[J]. 电子与信息学报, 2025, 47(8): 2584–2597. doi: 10.11999/JEIT250161. ZHENG Qinghe, LIU Fanglin, YU Lisu, et al. An improved modulation recognition method based on hybrid Kolmogorov-Arnold convolutional neural network[J]. Journal of Electronics & Information Technology, 2025, 47(8): 2584–2597. doi: 10.11999/JEIT250161.
[26]	Blealtan. An efficient implementation of Kolmogorov-Arnold network[EB/OL]. https://github.com/Blealtan/efficient-kan, 2024.
[27]	ZHANG Fan and ZHANG Xin. GraphKAN: Enhancing feature extraction with graph Kolmogorov Arnold networks[J]. arXiv: 2406.13597, 2024. doi: 10.48550/arXiv.2406.13597.
[28]	GAL Y and GHAHRAMANI Z. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning[C]. Proceedings of the 33nd International Conference on Machine Learning, New York, USA, 2016: 1050–1059.
[29]	SZEGEDY C, ZAREMBA W, SUTSKEVER I, et al. Intriguing properties of neural networks[C]. 2nd International Conference on Learning Representations, Banff, Canada, 2014.
[30]	PHAM H, DAI Zihang, XIE Qizhe, et al. Meta pseudo labels[C]. IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, USA, 2021: 11552–11563. doi: 10.1109/CVPR46437.2021.01139.
[31]	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, USA, 2017: 6000–6010.