A Meta-learning Knowledge Reasoning Framework Combining Semantic Path and Language Model

DUAN Li; FENG Haojun; ZHANG Biying; LIU Jiangzhou; LIU Haichao

doi:10.11999/JEIT211034

Volume 44 Issue 12

Dec. 2022

Turn off MathJax

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2022 > 44(12): 4376-4383

DUAN Li, FENG Haojun, ZHANG Biying, LIU Jiangzhou, LIU Haichao. A Meta-learning Knowledge Reasoning Framework Combining Semantic Path and Language Model[J]. Journal of Electronics & Information Technology, 2022, 44(12): 4376-4383. doi: 10.11999/JEIT211034

Citation:

DUAN Li, FENG Haojun, ZHANG Biying, LIU Jiangzhou, LIU Haichao. A Meta-learning Knowledge Reasoning Framework Combining Semantic Path and Language Model[J]. Journal of Electronics & Information Technology, 2022, 44(12): 4376-4383. doi: 10.11999/JEIT211034

Citation:

PDF( 4104 KB)

A Meta-learning Knowledge Reasoning Framework Combining Semantic Path and Language Model

doi: 10.11999/JEIT211034

1.
College of Electronic Engineering, Naval University of Engineering, Wuhan 430033, China
2.
Team 91202, Chinese People's Liberation Army, Huludao 125004, China

Received Date: 2021-09-27
Accepted Date: 2021-12-29
Rev Recd Date: 2021-12-17

Available Online: 2022-01-13

Publish Date: 2022-12-16

Abstract

Abstract

In order to solve the problems that traditional knowledge reasoning methods can not combine computing power and interpretability, and it is difficult to learn quickly in few-shot scenarios, a Model-Agnostic Meta-Learning (MAML) reasoning framework is proposed in this paper, which combines semantic path and Bidirectional Encoder Representations for Transformers (BERT), and consists of two stages: base-training and meta-training. In base-training stage, the graph reasoning instances is represented by semantic path and BERT model, which is used to calculate the link probability and save reasoning experience offline by fine-tuning. In meta-training stage, the gradient meta-information based on the base-training process of multiple relations is obtained by this framework, which realizes the initial weight optimization, and completes the rapid learning of knowledge under few-shot. Experiments show that better performance in link prediction and fact prediction can be achieved by the base-training reasoning framework, and fast convergence of some few-shot reasoning problems can be achieved by the meta-learning framework.
- Knowledge reasoning,
- Semantic path,
- Bidirectional Encoder Representations for Transformers (BERT),
- Model-Agnostic Meta-Learning (MAML)

FullText(HTML)

References(19)

References

[1]	马忠贵, 倪润宇, 余开航. 知识图谱的最新进展、关键技术和挑战[J]. 工程科学学报, 2020, 42(10): 1254–1266. doi: 10.13374/j.issn2095-9389.2020.02.28.001 MA Zhonggui, NI Runyu, and YU Kaihang. Recent advances, key techniques and future challenges of knowledge graph[J]. Chinese Journal of Engineering, 2020, 42(10): 1254–1266. doi: 10.13374/j.issn2095-9389.2020.02.28.001
[2]	官赛萍, 靳小龙, 贾岩涛, 等. 面向知识图谱的知识推理研究进展[J]. 软件学报, 2018, 29(10): 2966–2994. doi: 10.13328/j.cnki.jos.005551 GUAN Saiping, JIN Xiaolong, JIA Yantao, et al. Knowledge reasoning over knowledge graph: A survey[J]. Journal of Software, 2018, 29(10): 2966–2994. doi: 10.13328/j.cnki.jos.005551
[3]	LAO Ni and COHEN W W. Relational retrieval using a combination of path-constrained random walks[J]. Machine Learning, 2010, 81(1): 53–67. doi: 10.1007/s10994-010-5205-8
[4]	YANG Fan, YANG Zhilin, and COHEN W W. Differentiable learning of logical rules for knowledge base completion[J]. arXiv: 1702.08367, 2017.
[5]	康世泽, 吉立新, 张建朋. 一种基于图注意力网络的异质信息网络表示学习框架[J]. 电子与信息学报, 2021, 43(4): 915–922. doi: 10.11999/JEIT200034 KANG Shize, JI Lixin, and ZHANG Jianpeng. Heterogeneous information network representation learning framework based on graph attention network[J]. Journal of Electronics &Information Technology, 2021, 43(4): 915–922. doi: 10.11999/JEIT200034
[6]	刘藤, 陈恒, 李冠宇. 联合FOL规则的知识图谱表示学习方法[J]. 计算机工程与应用, 2021, 57(4): 100–107. doi: 10.3778/j.issn.1002-8331.1911-0436 LIU Teng, CHEN Heng, and LI Guanyu. Knowledge graph representation learning method jointing FOL rules[J]. Computer Engineering and Applications, 2021, 57(4): 100–107. doi: 10.3778/j.issn.1002-8331.1911-0436
[7]	ZHANG Linli, LI Dewei, XI Yugeng, et al. Reinforcement learning with actor-critic for knowledge graph reasoning[J]. Science China Information Sciences, 2020, 63(6): 169101. doi: 10.1007/s11432-018-9820-3
[8]	WANG Heng, LI Shuangyin, PAN Rong, et al. Incorporating graph attention mechanism into knowledge graph reasoning based on deep reinforcement learning[C]. The 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, 2019: 2623–2631.
[9]	陈海旭, 周强, 刘学军. 一种结合路径信息和嵌入模型的知识推理方法[J]. 小型微型计算机系统, 2020, 41(6): 1147–1151. doi: 10.3969/j.issn.1000-1220.2020.06.005 CHEN Haixu, ZHOU Qiang, and LIU Xuejun. Knowledge graph reasoning combining path information and embedding model[J]. Journal of Chinese Computer Systems, 2020, 41(6): 1147–1151. doi: 10.3969/j.issn.1000-1220.2020.06.005
[10]	LÜ Xin, GU Yuxian, HAN Xu, et al. Adapting meta knowledge graph information for multi-hop reasoning over few-shot relations[C]. The 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, 2019: 3376–3381.
[11]	CHEN Mingyang, ZHANG Wen, ZHANG Wei, et al. Meta relational learning for few-shot link prediction in knowledge graphs[C]. The 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, 2019: 4217–4226.
[12]	DEVLIN J, CHANG Mingwei, LEE K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding[C]. 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, USA, 2019.
[13]	FINN C, ABBEEL P, and LEVINE S. Model-agnostic meta-learning for fast adaptation of deep networks[C]. The 34th International Conference on Machine Learning, Sydney, Australia, 2017: 1126–1135.
[14]	YANG Zhilin, DAI Zihang, YANG Yiming, et al. XLNet: Generalized autoregressive pretraining for language understanding[C]. The 33rd Conference on Neural Information Processing Systems, Vancouver, Canada, 2019.
[15]	SUN Yu, WANG Shuohuan, LI Yukun, et al. ERNIE: Enhanced representation through knowledge integration[J]. arXiv: 1904.09223, 2019.
[16]	SUN Chi, QIU Xipeng, XU Yige, et al. How to fine-tune BERT for text classification?[C]. The 18th China National Conference on Chinese Computational Linguistics, Kunming, China, 2019: 194–206.
[17]	任进军, 王宁. 人工神经网络中损失函数的研究[J]. 甘肃高师学报, 2018, 23(2): 61–63. doi: 10.3969/j.issn.1008-9020.2018.02.019 REN Jinjun and WANG Ning. Research on cost function in artificial neural network[J]. Journal of Gansu Normal Colleges, 2018, 23(2): 61–63. doi: 10.3969/j.issn.1008-9020.2018.02.019
[18]	BORDES A, USUNIER N, GARCIA-DURÁN A, et al. Translating embeddings for modeling multi-relational data[C]. The 26th International Conference on Neural Information Processing Systems, Lake Tahoe, USA, 2013: 2787–2795.
[19]	SCHLICHTKRULL M, KIPF T N, BLOEM P, et al. Modeling relational data with graph convolutional networks[C]. The 15th International Conference on The Semantic Web, Heraklion, Greece, 2018: 593–607.