A Knowledge Distillation Framework for Hypergraph Neural Networks with Fast Inference

LI Junzheng; YU Hongtao; HUANG Ruiyang; JIANG Haocong; LIU Shuo; YANG Suchang

doi:10.11999/JEIT260694

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2025 >

LI Junzheng, YU Hongtao, HUANG Ruiyang, JIANG Haocong, LIU Shuo, YANG Suchang. A Knowledge Distillation Framework for Hypergraph Neural Networks with Fast Inference[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260694

Citation:

LI Junzheng, YU Hongtao, HUANG Ruiyang, JIANG Haocong, LIU Shuo, YANG Suchang. A Knowledge Distillation Framework for Hypergraph Neural Networks with Fast Inference[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260694

Citation:

PDF( 2040 KB)

A Knowledge Distillation Framework for Hypergraph Neural Networks with Fast Inference

doi: 10.11999/JEIT260694 cstr: 32379.14.JEIT260694

Information Engineering University, Zhengzhou 450000, China

Received Date: 2026-05-28
Accepted Date: 2026-08-24
Rev Recd Date: 2026-08-18

Available Online: 2026-09-01

Abstract

Abstract

Objective HyperGraph Neural Networks (HGNNs) have received increasing attention because of their ability to model higher-order relationships among entities. However, their high computational cost and low inference efficiency limit their deployment in large-scale applications. Existing knowledge distillation methods that transfer knowledge from HGNNs to Multi-Layer Perceptrons (MLPs) have limited interpretability and accuracy. They also suffer from information loss in softmax-based soft labels and do not account for differences in node reliability. To address these limitations, a hypergraph knowledge distillation framework, Distill Hypergraph neural network to Kolmogorov-Arnold Network (DH2KAN), is proposed for fast inference. Methods DH2KAN consists of three main modules (Fig. 2). First, a Kolmogorov-Arnold Network (KAN) is adopted as the student model instead of an MLP. Learnable spline-based univariate functions replace fixed activation functions and linear weights to improve the fitting ability and interpretability of the student model. Second, a representation similarity distillation mechanism is introduced to directly align the pre-logit representations of the HGNN and KAN. This alignment reduces the information loss caused by softmax normalization and preserves higher-order structural knowledge in the hypergraph. Third, a high-reliability node-aware distillation method is introduced (Fig. 3). Node reliability is quantified from changes in information entropy under noise perturbations. High-reliability nodes that are less sensitive to noise are selected, and their soft labels are used as additional supervision to improve the quality of distilled knowledge. Results and Discussions DH2KAN is evaluated under both transductive learning (Table 2) and production learning (Table 3) settings. Under transductive learning, its average accuracy is 10.53 percentage points higher than that of the baseline KAN student model, 1.73 percentage points higher than that of the teacher HGNN, and 1.20 percentage points higher than that of the MLP-based distillation baseline. These results confirm the effectiveness of knowledge transfer from the HGNN to the KAN and show that the proposed method achieves higher inference accuracy than conventional MLP-based distillation methods. In addition, DH2KAN is particularly well suited to feature-dominated hypergraphs and remains robust on hypergraphs with sparse structures and noisy nodes. Conclusions DH2KAN is proposed to accelerate HGNN inference in large-scale, low-latency applications. Knowledge distillation narrows the performance gap between the KAN and HGNN and removes dependence on the hypergraph structure during inference, thereby enabling fast, low-complexity prediction. Representation similarity distillation and high-reliability node-aware distillation use the teacher model’s pre-logit representations and the soft labels of high-reliability nodes as additional supervision, respectively. These mechanisms enable effective transfer of task-relevant knowledge to the student model and support the practical deployment of DH2KAN in large-scale, low-latency scenarios.
- HyperGraph Neural Network (HGNN),
- Kolmogorov-Arnold Network (KAN),
- Knowledge distillation,
- Fast inference

FullText(HTML)

References(31)

References

[1]	谢丽霞, 史镜琛, 杨宏宇, 等. 基于图神经网络模型校准的成员推理攻击[J]. 电子与信息学报, 2025, 47(3): 780–791. doi: 10.11999/JEIT240477. XIE Lixia, SHI Jingchen, YANG Hongyu, et al. Membership inference attacks based on graph neural network model calibration[J]. Journal of Electronics & Information Technology, 2025, 47(3): 780–791. doi: 10.11999/JEIT240477.
[2]	吴翼腾, 刘伟, 于洪涛, 等. 基于局部影响分析模型的图神经网络对抗攻击[J]. 电子与信息学报, 2022, 44(7): 2576–2583. doi: 10.11999/JEIT210448. WU Yiteng, LIU Wei, YU Hongtao, et al. Adversarial attacks on graph neural network based on local influence analysis model[J]. Journal of Electronics & Information Technology, 2022, 44(7): 2576–2583. doi: 10.11999/JEIT210448.
[3]	GAO Yue, ZHANG Zizhao, LIN Haojie, et al. Hypergraph learning: Methods and practices[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(5): 2548–2566. doi: 10.1109/TPAMI.2020.3039374.
[4]	TIAN Yijun, PEI Shichao, ZHANG Xiangliang, et al. Knowledge distillation on graphs: A survey[J]. ACM Computing Surveys, 2025, 57(8): 189. doi: 10.1145/3711121.
[5]	FENG Yifan, LUO Yihe, YING Shihui, et al. LightHGNN: Distilling hypergraph neural networks into MLPs for 100x faster inference[C]. The Twelfth International Conference on Learning Representations, Vienna, Austria, 2024:29822-29842.
[6]	LIU Ziming, WANG Yixuan, VAIDYA S, et al. KAN: Kolmogorov–Arnold networks[C]. The Thirteenth International Conference on Learning Representations, Singapore, Singapore, 2025: 70367–70413.
[7]	LIAO Sihao, XIE Liang, DU Yuanchuang, et al. Stock trend prediction based on dynamic hypergraph spatio-temporal network[J]. Applied Soft Computing, 2024, 154: 111329. doi: 10.1016/J.ASOC.2024.111329.
[8]	KIM S, LEE S Y, GAO Yue, et al. A survey on hypergraph neural networks: An in-depth and step-by-step guide[C]. The 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Barcelona, Spain, 2024: 6534–6544.
[9]	BERGE C. Hypergraphs: Combinatorics of Finite Sets[M]. Amsterdam: Elsevier, 1989.
[10]	FENG Yifan, YOU Haoxuan, ZHANG Zizhao, et al. Hypergraph neural networks[C]. The 33rd AAAI Conference on Artificial Intelligence, Honolulu, USA, 2019: 3558–3565.
[11]	GAO Yue, FENG Yifan, JI Shuyi, et al. HGNN⁺: General hypergraph neural networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(3): 3181–3199. doi: 10.1109/TPAMI.2022.3182052.
[12]	YADATI N, NIMISHAKAVI M, YADAV P, et al. HyperGCN: A new method of training graph convolutional networks on hypergraphs[C]. The 33rd International Conference on Neural Information Processing Systems, Vancouver, Canada, 2019: 135.
[13]	DONG Yihe, SAWIN W, and BENGIO Y. HNHN: Hypergraph networks with hyperedge neurons[J]. arXiv: 2006.12278, 2020. doi: 10.48550/arXiv.2006.12278.
[14]	JIANG Jianwen, WEI Yuxuan, FENG Yifan, et al. Dynamic hypergraph neural networks[C]. The Twenty-Eighth International Joint Conference on Artificial Intelligence, Macao, China, 2019: 2635–2641. doi: 10.24963/ijcai.2019/366.
[15]	BAI Song, ZHANG Feihu, and TORR P H S. Hypergraph convolution and hypergraph attention[J]. Pattern Recognition, 2021, 110: 107637. doi: 10.1016/j.patcog.2020.107637.
[16]	JO J, BAEK J, LEE S, et al. Edge representation learning with hypergraphs[C]. The 35th International Conference on Neural Information Processing Systems, 2021: 577.
[17]	HUANG Jing and YANG Jie. UniGNN: A unified framework for graph and hypergraph neural networks[C]. The Thirtieth International Joint Conference on Artificial Intelligence, Montreal, Canada, 2021: 2563–2569. doi: 10.24963/IJCAI.2021/353.
[18]	CHIEN E, PAN Chao, PENG Jianhao, et al. You are AllSet: A multiset function framework for hypergraph neural networks[C]. The Tenth International Conference on Learning Representations, 2022.
[19]	HINTON G, VINYALS O, and DEAN J. Distilling the knowledge in a neural network[J]. arXiv: 1503.02531, 2015. doi: 10.48550/arXiv.1503.02531.
[20]	罗一畅, 齐析屿, 张博锐, 等. 分割一切模型的轻量化研究综述[J]. 电子与信息学报, 2026, 48(2): 713–731. doi: 10.11999/JEIT250894. LUO Yichang, QI Xiyu, ZHANG Borui, et al. A survey of lightweight techniques for segment anything model[J]. Journal of Electronics & Information Technology, 2026, 48(2): 713–731. doi: 10.11999/JEIT250894.
[21]	ZHANG Shichang, LIU Yozen, SUN Yizhou, et al. Graph-less neural networks: Teaching old MLPs new tricks via distillation[C]. The Tenth International Conference on Learning Representations, 2022.
[22]	YANG Cheng, LIU Jiawei, and SHI Chuan. Extract the knowledge of graph neural networks and go beyond it: An effective knowledge distillation framework[C]. The Web Conference 2021, Ljubljana, Slovenia, 2021: 1227–1237. doi: 10.1145/3442381.3450068.
[23]	WU Lirong, LIN Haitao, HUANG Yufei, et al. Quantifying the knowledge in GNNs for reliable distillation into MLPs[C]. The 40th International Conference on Machine Learning, Honolulu, USA, 2023: 37571–37581.
[24]	LIU Ziming, MA Pingchuan, WANG Yixuan, et al. KAN 2.0: Kolmogorov-Arnold networks meet science[J]. arXiv: 2408.10205, 2024. doi: 10.48550/arxiv.2408.10205.
[25]	郑庆河, 刘方霖, 余礼苏, 等. 基于改进Kolmogorov-Arnold混合卷积神经网络的调制识别方法[J]. 电子与信息学报, 2025, 47(8): 2584–2597. doi: 10.11999/JEIT250161. ZHENG Qinghe, LIU Fanglin, YU Lisu, et al. An improved modulation recognition method based on hybrid Kolmogorov-Arnold convolutional neural network[J]. Journal of Electronics & Information Technology, 2025, 47(8): 2584–2597. doi: 10.11999/JEIT250161.
[26]	Blealtan. An efficient implementation of Kolmogorov-Arnold network[EB/OL]. https://github.com/Blealtan/efficient-kan, 2024.
[27]	ZHANG Fan and ZHANG Xin. GraphKAN: Enhancing feature extraction with graph Kolmogorov Arnold networks[J]. arXiv: 2406.13597, 2024. doi: 10.48550/arXiv.2406.13597.
[28]	GAL Y and GHAHRAMANI Z. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning[C]. The 33nd International Conference on Machine Learning, New York, USA, 2016: 1050–1059.
[29]	SZEGEDY C, ZAREMBA W, SUTSKEVER I, et al. Intriguing properties of neural networks[C]. 2nd International Conference on Learning Representations, Banff, Canada, 2014.
[30]	PHAM H, DAI Zihang, XIE Qizhe, et al. Meta pseudo labels[C]. IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, USA, 2021: 11552–11563. doi: 10.1109/CVPR46437.2021.01139.
[31]	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]. The 31st International Conference on Neural Information Processing Systems, Long Beach, USA, 2017: 6000–6010.