Biomedical Word Sense Disambiguation via Contrastive Learning with Feature and Sample Enhancement
-
摘要: 词义消歧(WSD)是生物医学文本理解与信息挖掘的核心技术，广泛应用于医学文献分析、临床数据处理等领域。为解决现有模型在生物医学场景中面临的术语语义易受噪声干扰、细粒度语义类区分难、小样本泛化不足等问题，该文提出了一种将生物医学术语中的词形、词性、语义类作为核心消歧特征，输入至Electra、mDeBERTa与T5三路模型中来获取领域适配的动态词向量，使用Focal loss和Margin loss组合的复合损失函数三路并行架构。通过卡方统计量引导的注意力机制，来获取更加具有代表性的特征，以抑制冗余噪声干扰。训练过程中采用第一阶段挖掘困难样本第二阶段克服困难样本的困难样本挖掘策略。引入基于生物医学术语裁剪的对比学习方法，生成两个语义相同的视图进行对比与学习，来强化特征的语义鲁棒性与区分性。实验结果证明，该方法能够显著地提升模型对近义语义类的区分能力，并增强模型在噪声高、少样本场景下的鲁棒性与泛化能力，表现出更优秀的生物医学词义消歧性能。Abstract:
Objective With the rapid growth of biomedical literature, biomedical word sense disambiguation (WSD) has become essential for medical text mining and clinical data analysis. However, existing methods suffer from semantic noise, fine-grained category discrimination, and limited generalization in low-resource scenarios. This study proposes a three-branch parallel WSD framework with contrastive learning, integrating multi-pretrained models, chi-square attention, Focal+Margin hybrid loss, and hard sample mining. The proposed method improves semantic representation, robustness, and discrimination ability, providing an effective solution for biomedical semantic mining. Methods The proposed framework integrates Electra, mDeBERTa, and Flan-T5 to extract complementary contextual features from biomedical terms. A chi-square attention module is designed to select representative features, while a Focal+Margin hybrid loss improves discrimination under class imbalance. In addition, a two-stage hard sample mining strategy and a core-term constrained contrastive learning mechanism are introduced to enhance the learning of difficult samples and semantic boundaries. Results and Discussions The proposed framework integrates chi-square attention and contrastive learning into a three-branch parallel architecture for biomedical WSD. Experiments on the MSH dataset show that the proposed model achieves an accuracy of 95.27%, outperforming the state-of-the-art Neural Concept Embeddings by 0.93%. Ablation studies on contrastive parameters further demonstrate its effectiveness in enhancing semantic discrimination and generalization ability. The model also reduces confusion among similar biomedical terms, achieving an F1-score of 92.1% on minority semantic classes. Conclusions This study proposes a three-branch parallel contrastive learning framework for biomedical WSD in complex semantic environments. The framework integrates Electra, mDeBERTa, and FT5 to capture complementary semantic features, while combining chi-square attention, contrastive learning, and two-stage hard sample training to enhance feature discrimination and robustness. Experimental results demonstrate that the proposed method effectively improves disambiguation performance and reduces confusion among semantically similar biomedical concepts. However, this study is limited to monolingual English biomedical texts. Future work will explore multilingual biomedical corpora and integrate domain-specific knowledge graphs to further improve semantic representation. -
Key words:
- Word sense disambiguation /
- Biomedicine /
- Contrastive learning /
- Disambiguation feature /
- Loss function
-
表 1 对比学习算法对模型性能的影响(%)
算法 Accuracy Precision Recall F1 Negative samples 95.27 90.33 85.49 87.84 Cluster 94.49 87.61 84.36 85.95 Asymmetric Network 94.14 85.71 84.87 85.29 Memory Bank 91.19 83.92 69.24 75.87 Adversarial Examples 93.37 85.87 80.04 82.85 
下载: 导出CSV
表 2 对比损失对模型性能的影响(%)
对比损失 Accuracy Precision Recall F1 NT-Xent 95.27 90.33 85.49 87.84 Triplet 94.61 88.66 83.74 86.13 SupCon 94.42 86.77 85.07 85.91 Barlow Twins 93.20 87.49 76.99 81.91 Circle 94.01 87.38 81.89 84.55
下载: 导出CSV
表 3 第二视图对模型性能的影响(%)
方法 Accuracy Precision Recall F1 synonym 94.16 87.31 82.82 85.01 mask 92.9 85.58 77.57 81.38 crop 95.27 90.33 85.49 87.84 drop-modality 93.83 87.92 80.14 83.85
下载: 导出CSV
表 4 困难样本集构建方法对模型性能的影响(%)
所基于的方法 Accuracy Precision Recall F1 训练损失 94.77 88.56 84.77 86.63 不确定性 94.12 87.86 81.94 84.79 训练损失及不确定性 95.27 90.33 85.49 87.84 自步学习 94.20 88.98 81.00 84.81 在线困难样本挖掘 94.50 87.11 85.10 86.09 随机样本选取 93.93 86.75 82.19 84.41
下载: 导出CSV
表 5 α对模型性能的影响(%)
α Accuracy Precision Recall F1 0 93.67 87.39 79.88 83.46 0.2 94.11 89.02 80.49 84.54 0.4 94.22 88.30 81.94 85.00 0.6 95.27 90.33 85.49 87.84 0.8 94.19 88.89 81.11 84.82 1.0 94.27 88.50 81.99 85.12
下载: 导出CSV
表 6 消融实验(%)
模型 Accuracy Precision Recall F1 EM 85.20 65.13 55.92 60.18 EMw 88.39 72.17 68.26 70.16 EMwT 91.56 85.04 70.16 76.89 EMwT-H 92.47 83.89 77.16 80.39 EMwT-HC 93.95 87.33 81.58 84.36 EMwT-HCL 94.16 87.31 82.82 85.01 EMwT-HCL-Cl 95.27 90.33 85.49 87.84
下载: 导出CSV
表 7 对比实验(%)
模型 Accuracy Electra 70.89 mDeBERTa 69.90 FT5 72.70 Bert 78.27 Word-Concept Model 89.10 Attention Neural Network 91.38 Attention cct-T 93.94 Neural Concept Embeddings 94.34 EMwT-HCL-Cl 95.27
下载: 导出CSV
表 8 参数量及时间复杂度
模型 参数量 (M) 时间复杂度 (Big-O) Electra 110 $ O(L\cdot H_{E}^{2}) $ mDeBERTa 180 $ O(L\cdot H_{M}^{2}) $ FT5 250 $ O(L\cdot H_{T}^{2}) $ 卡方注意力 0.77 $ O(L\cdot H\cdot D) $ 困难样本挖掘 0 $ O(N\cdot {f}_{\text{forward}}) $ 对比学习 0.34 $ O(F\cdot H) $ EMwT-HCL-Cl ≈541.1 M $ O(L(H_{E}^{2}+H_{M}^{2}+H_{T}^{2})+LDH+FH) $
下载: 导出CSV
-
[1] 李俊辉, 侯兴松. 基于伪监督注意力短期记忆与多尺度去伪影网络的图像分块压缩感知[J]. 电子与信息学报, 2024, 46(2): 472–480. doi: 10.11999/JEIT231069.LI Junhui and HOU Xingsong. Pseudo supervised attention short-term memory and multi-scale deartifacting network based on image block compressed sensing[J]. Journal of Electronics & Information Technology, 2024, 46(2): 472–480. doi: 10.11999/JEIT231069. [2] 杨春玲, 梁梓文. 静态与动态域先验增强的两阶段视频压缩感知重构网络[J]. 电子与信息学报, 2024, 46(11): 4247–4258. doi: 10.11999/JEIT240295.YANG Chunling and LIANG Ziwen. Static and dynamic-domain prior enhancement two-stage video compressed sensing reconstruction network[J]. Journal of Electronics & Information Technology, 2024, 46(11): 4247–4258. doi: 10.11999/JEIT240295. [3] SUNG S F, HU Yahan, and CHEN Chongyan. Disambiguating clinical abbreviations by one-to-all classification: Algorithm development and validation study[J]. JMIR Medical Informatics, 2024, 12: e56955. doi: 10.2196/56955. [4] KADDOURA S and NASSAR R. EnhancedBERT: A feature-rich ensemble model for Arabic word sense disambiguation with statistical analysis and optimized data collection[J]. Journal of King Saud University-Computer and Information Sciences, 2024, 36(1): 101911. doi: 10.1016/j.jksuci.2023.101911. [5] CAO Yukun, JIN Chengkun, TANG Yijia, et al. Word sense disambiguation combining knowledge graph and text hierarchical structure[J]. ACM Transactions on Asian and Low-Resource Language Information Processing, 2024, 23(12): 161. doi: 10.1145/3677524. [6] 张春祥, 孙颖, 高可心, 等. 结合预训练模型的双向门控图卷积对抗词义消歧[J]. 电子与信息学报, 2025, 47(11): 4549–4559. doi: 10.11999/JEIT250386.ZHANG Chunxiang, SUN Ying, GAO Kexin, et al. Combine the pre-trained model with bidirectional gated recurrent units and graph convolutional network for adversarial word sense disambiguation[J]. Journal of Electronics & Information Technology, 2025, 47(11): 4549–4559. doi: 10.11999/JEIT250386. [7] ION R, PĂIȘ V, MITITELU V B, et al. Unsupervised word sense disambiguation using transformer’s attention mechanism[J]. Machine Learning and Knowledge Extraction, 2025, 7(1): 10. doi: 10.3390/make7010010. [8] RANSING R and GULATI A. Unsupervised word sense disambiguation for Marathi language using word embeddings[J]. International Journal of Intelligent Systems and Applications in Engineering, 2024, 12(3): 1374–1380. [9] PADWAD H, KESWANI G, BISEN W, et al. Leveraging contextual factors for word sense disambiguation in Hindi language[J]. International Journal of Intelligent Systems and Applications in Engineering, 2024, 12(12s): 129–136. [10] RANSING R and GULATI A. Marathi word sense disambiguation through unsupervised k-means clustering[J]. Engineering, Technology & Applied Science Research, 2025, 15(3): 22837–22843. doi: 10.48084/etasr.9975. [11] MARTIN D I, BERRY M W, and MARTIN J C. Semantic unsupervised learning for word sense disambiguation[M]. BERRY M W, MOHAMED A, and YAP B W. Supervised and Unsupervised Learning for Data Science. Cham, Switzerland: Springer, 2019: 101–120. doi: 10.1007/978-3-030-22475-2_6. [12] RAHMANI S, FAKHRAHMAD S M, and SADREDDINI M H. Co-occurrence graph-based context adaptation: A new unsupervised approach to word sense disambiguation[J]. Digital Scholarship in the Humanities, 2021, 36(2): 449–471. doi: 10.1093/llc/fqz048. [13] LU Wenpeng, MENG Fanqing, WANG Shoujin, et al. Graph-based Chinese word sense disambiguation with multi-knowledge integration[J]. CMC-Computers, Materials & Continua, 2019, 61(1): 197–212. doi: 10.32604/cmc.2019.06068. [14] BLEVINS T and ZETTLEMOYER L. Moving down the long tail of word sense disambiguation with Gloss Informed Bi-encoders[C]. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020: 1006–1017. doi: 10.18653/v1/2020.acl-main.95.(查阅网上资料,未找到出版地信息,请补充). [15] ANTUNES R and MATOS S. Supervised learning and knowledge-based approaches applied to biomedical word sense disambiguation[J]. Journal of Integrative Bioinformatics, 2017, 14(4): 20170051. doi: 10.1515/jib-2017-0051. [16] ALSAEEDAN W, MENAI M E B, and AL-AHMADI S. A hybrid genetic-ant colony optimization algorithm for the word sense disambiguation problem[J]. Information Sciences, 2017, 417: 20–38. doi: 10.1016/j.ins.2017.07.002. [17] SARROUTI M and OUATIK EL ALAOUI S. SemBioNLQA: A semantic biomedical question answering system for retrieving exact and ideal answers to natural language questions[J]. Artificial Intelligence in Medicine, 2020, 102: 101767. doi: 10.1016/j.artmed.2019.101767. [18] CHEN Fangyi, ZHANG Gongbo, CHEN Si, et al. Clinical note structural knowledge improves word sense disambiguation[J]. AMIA Joint Summits on Translational Science Proceedings, 2024, 2024: 515–524. [19] RAIS M and LACHKAR A. An empirical study of word sense disambiguation for biomedical information retrieval system[C]. Proceedings of the 6th International Conference on Bioinformatics and Biomedical Engineering, Granada, Spain, 2018: 314–326. doi: 10.1007/978-3-319-78723-7_27. [20] PASHUK A V, GURINOVICH A B, VOLOROVA N A, et al. Analysis of the methods of word sense disambiguation in the biomedical domain[J]. Doklady BGUIR, 2019, 75(5): 60–65. doi: 10.35596/1729-7648-2019-123-5-60-65. [21] YAP B P, KOH A, and CHNG E S. Adapting BERT for word sense disambiguation with gloss selection objective and example sentences[C]. Findings of the Association for Computational Linguistics, 2020: 41–46. doi: 10.18653/v1/2020.findings-emnlp.4. (查阅网上资料,未找到出版地信息,请补充). [22] KWON S, OH D, and KO Y. Word sense disambiguation based on context selection using knowledge-based word similarity[J]. Information Processing & Management, 2021, 58(4): 102551. doi: 10.1016/j.ipm.2021.102551. [23] ALMOUSA M, BENLAMRI R, and KHOURY R. A novel word sense disambiguation approach using WordNet knowledge graph[J]. Computer Speech & Language, 2022, 74: 101337. doi: 10.1016/j.csl.2021.101337. [24] CHEN Ting, KORNBLITH S, NOROUZI M, et al. A simple framework for contrastive learning of visual representations[C]. 37th International Conference on Machine Learning, 2020: 1597–1607. (查阅网上资料, 未找到出版地信息, 请补充). [25] CARON M, MISRA I, MAIRAL J, et al. Unsupervised learning of visual features by contrasting cluster assignments[C]. Proceedings of the 34th International Conference on Neural Information Processing Systems, Vancouver, Canada, 2020: 831. [26] GRILL J B, STRUB F, ALTCHÉ F, et al. Bootstrap your own latent a new approach to self-supervised learning[C]. Proceedings of the 34th International Conference on Neural Information Processing Systems, Vancouver, Canada, 2020: 1786. [27] WU Zhirong, XIONG Yuanjun, YU S X, et al. Unsupervised feature learning via non-parametric instance discrimination[C]. Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018: 3733–3742. doi: 10.1109/CVPR.2018.00393. [28] HU Qianjiang, WANG Xiao, HU Wei, et al. AdCo: Adversarial contrast for efficient learning of unsupervised representations from self-trained negative adversaries[C]. Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, USA, 2021: 1074–1083. doi: 10.1109/CVPR46437.2021.00113. [29] YEPES A J and BERLANGA R. Knowledge based word-concept model estimation and refinement for biomedical text mining[J]. Journal of Biomedical Informatics, 2015, 53: 300–307. doi: 10.1016/j.jbi.2014.11.015. [30] ZHANG Linfeng, CHEN Xin, ZHANG Junbo, et al. Contrastive deep supervision[C]. 17th European Conference on Computer Vision, Tel Aviv, Israel, 2022: 1–19. doi: 10.1007/978-3-031-19809-0_1. [31] SABBIR A K M, JIMENO-YEPES A, and KAVULURU R. Knowledge-based biomedical word sense disambiguation with neural concept embeddings[C]. 2017 IEEE 17th International Conference on Bioinformatics and Bioengineering, Washington, USA, 2017: 163–170. doi: 10.1109/BIBE.2017.00-61. [32] ZHANG Canlin, BIŚ D, LIU Xiuwen, et al. Biomedical word sense disambiguation with bidirectional long short-term memory and attention-based neural networks[J]. BMC Bioinformatics, 2019, 20(S16): 502. doi: 10.1186/s12859-019-3079-8. [33] CHUNG H W, HOU Le, LONGPRE S, et al. Scaling instruction-finetuned language models[J]. The Journal of Machine Learning Research, 2024, 25(1): 70. -
下载:
图(5) / 表(8)
计量
- 文章访问数: 13
- HTML全文浏览量: 1
- PDF下载量: 2
- 被引次数: 0
