一种面向数据稀缺场景的文本分类模型训练方法

徐森; 邓驹骠; 徐秀芳; 姚善良; 卞学胜; 贲晛烨

doi:10.11999/JEIT260365

一种面向数据稀缺场景的文本分类模型训练方法

doi: 10.11999/JEIT260365 cstr: 32379.14.JEIT260365

徐森^{1, 2, 3, ,},
邓驹骠^{1, 2,},
徐秀芳^{1, 2},
姚善良^{1, 2},
卞学胜^{1, 2},
贲晛烨^{3, 4}

1.
盐城工学院信息工程学院盐城 224051
2.
海上风电系统多模态感知与智能控制江苏省产业技术工程化中心盐城 224051
3.
哈尔滨工程大学信息与通信工程学院哈尔滨 150001
4.
山东大学信息科学与工程学院济南 266237

基金项目: 国家自然科学基金(62676350, 52602749, 62301473, 62322111), 江苏高校“青蓝工程”，盐城市产业创新科技支撑(工业)专项(YCBG2025201), 黑龙江省自然科学基金 (ZD2025F004)

详细信息

作者简介:
徐森：男，教授，研究方向为人工智能、计算机技术、数据挖掘与融合，xusen@ycit.cn

邓驹骠：男，硕士，研究方向为人工智能，2544150359@qq.com

徐秀芳：女，高级实验师，研究方向为人工智能、计算机技术

姚善良：男，讲师，研究方向为人工智能

卞学胜：男，讲师，研究方向为人工智能、数据挖掘与融合

贲晛烨：女，教授，研究方向为人工智能、计算机技术

通讯作者:
徐森　xusen@ycit.cn

1¹⁾https://huggingface.co/FacebookAI/roberta-base²⁾https://huggingface.co/distilbert/distilbert-base-uncased³⁾https://huggingface.co/albert/albert-base-v2⁴⁾https://huggingface.co/microsoft/deberta-base
中图分类号: TP181; TP391.1
计量
- 文章访问数: 124
- HTML全文浏览量: 24
- PDF下载量: 11
- 被引次数: 0
出版历程
- 收稿日期: 2026-03-30
- 修回日期: 2026-08-17
- 录用日期: 2026-08-17
- 网络出版日期: 2026-08-26

A Heterogeneous Multi-View Semantic Fusion Training Method for Text Classification in Low-Resource Scenarios

XU Sen^{1, 2, 3
, ,},
DENG Jubiao^{1, 2
,},
XU Xiufang^{1, 2},
YAO Shanliang^{1, 2},
BIAN Xuesheng^{1, 2},
BEN Xianye^{3, 4}

1.
School of Information Engineering, Yancheng Institute of Technology, Yancheng 224051, China
2.
Jiangsu Provincial Engineering Center for Offshore Wind Power Multi-Modal Perception and Intelligent Control, Yancheng 224051, China
3.
College of Information and Communication Engineering, Harbin Engineering University, Harbin 150001, China
4.
School of Information Science and Engineering, Shandong University, Qingdao 266237, China

Funds: The National Natural Science Foundation of China (62676350, 52602749, 62301473, 62322111), Jiangsu University Qing Lan Project, Yancheng Industrial Innovation Technology Support (Industrial) Special Program(YCBG2025201), The Natural Science Foundation of Heilongjiang Province (ZD2025F004)

摘要

摘要: 现有文本分类模型在标注数据稀缺场景下性能受限，其训练过程通常依赖单一视图的弱监督学习，易受噪声干扰，且难以挖掘难例样本在其他视图中的潜在判别信息。为解决上述问题，本文提出一种基于异构多视图语义融合的文本分类模型训练方法(HMVSF)。首先，在词频视图以双聚类一致性构建聚类一致样本集并进行一次轻量更新，提升伪监督可靠性；其次，开展主题建模，自适应选择主题数，生成硬标签与软分布，引导一次难例重训以充分挖掘其主题视图潜在特征；最后，两阶段共享同一编码器，在小规模标注集上统一微调，形成不依赖于特定的预训练起点的训练流程。在DBpedia、AG News、ISEAR、SMS Spam、Subjectivity、Polarity 6个公开数据集上，基于BERT分别设置两条初始化实验路线，实验结果表明，HMVSF整体优于代表性基线与单阶段变体，低标注预算下的提升尤为显著。
- 文本分类 /
- 多视图融合 /
- 文本聚类 /
- 预训练语言模型 /
- 中间训练
Abstract: Objective The increasing demand for text classification in specialized domains, such as finance, healthcare, and social media, is often hindered by the scarcity of labeled data. Low-resource or data-scarce scenarios significantly limit the effectiveness of conventional supervised learning and pre-trained language models, such as BERT, due to insufficient task-specific supervision. Existing methods that rely on single-view weak supervision or clustering often generate noisy pseudo-labels and fail to fully exploit the discriminative potential of hard-to-classify samples. Therefore, it is essential to develop a robust training method that leverages unlabeled data, mitigates noise sensitivity, and enhances semantic representation in low-label conditions. The primary objective of this research is to propose a text classification training framework specifically designed for data-scarce scenarios. Methods To address these challenges, a Heterogeneous Multi-View Semantic Fusion (HMVSF) training framework is introduced. HMVSF consists of three main stages: (1) Dual Clustering Consistency Screening (DCCS): Text samples are first represented in the word-frequency view using TF-IDF vectors. Two heterogeneous clustering algorithms, sIB and an improved K-means initialized via a genetic algorithm, are applied to the same feature space. Samples that are consistently assigned to intersecting clusters are selected as high-confidence pseudo-labeled instances. These samples are subsequently used for a lightweight supervised update of the shared encoder, ensuring reliable initialization for subsequent training. (2) Latent Dirichlet Allocation (LDA) Guided Hard-Sample Retraining: The remaining hard examples, which are not selected in the first stage, typically exhibit low discriminability in the word-frequency view but carry global semantic and topic-level information. LDA is employed to model the topic distribution of these samples. The optimal number of topics is adaptively determined by a combination of semantic coherence and statistical fitting criteria. Hard and soft pseudo-labels are generated based on the document-topic distribution, and the model undergoes targeted retraining using a combined cross-entropy and symmetric KL divergence loss to integrate topic-level semantic cues. (3) Downstream Fine-Tuning: Finally, the encoder is fine-tuned on a small labeled set by replacing temporary heads with a task-specific classification head. Standard cross-entropy is employed as the objective. The two-stage intermediate training is independent of the pre-training initialization, allowing HMVSF to be applied to both general-purpose BERT and masked language model (MLM)-based initializations, making it widely applicable. Results and Discussions The effectiveness of HMVSF is evaluated on six publicly available datasets, including DBpedia, AGNews, ISEAR, SMS Spam, Subjectivity, and Polarity, which cover both topic-based and non-topic-based text classification tasks. Two initialization routes are tested: Route A (BERT/BERT_IT:CLUST) and Route B (BERT_IT:MLM/BERT_IT:MLM+CLUST). Under extremely low label budgets (e.g., 64 labeled samples), CCS-BERT_IT:CLUST and CCS-BERT_IT:MLM+CLUST outperform baseline models and single-stage variants across datasets (Figs.2–3, Tables 2–3). For instance, in Route A, CCS-BERT_IT:CLUST achieves up to 9% accuracy improvement on DBpedia and ISEAR compared with BERT_IT:CLUST, while error rates decrease by 10–54% (Table 2). Normalized Mutual Information (NMI) and intra-cluster Euclidean distance analyses confirm that the CCS selection produces stable and compact clusters, ensuring reliable pseudo-labels (Tables 4–5). Ablation experiments show that the two-stage design combining CCS and LDA is superior to single-stage variants, demonstrating complementary advantages: CCS provides high-confidence core samples, and LDA captures latent semantic structures of hard examples (Table 7). Parameter sensitivity analysis reveals that the model is robust under a wide range of cluster numbers and LDA weight coefficients, achieving stable performance when the LDA weighting coefficient is set to 0.3 (Figs.4–5). HMVSF also maintains competitive performance as labeled data increase. Computational experiments on an NVIDIA A40 GPU demonstrate that the two-stage intermediate training introduces acceptable additional overhead while enhancing representation quality and overall accuracy. Conclusions This study presents a novel heterogeneous multi-view semantic fusion training method (HMVSF) for data-scarce text classification. By integrating cluster-consistent sample selection in the word-frequency view and LDA-guided hard-sample retraining, HMVSF effectively leverages unlabeled data, mitigates pseudo-label noise, and enriches semantic representations. Extensive experiments across multiple datasets confirm that HMVSF significantly improves classification accuracy under low-label conditions while remaining robust to pre-training initialization. The framework is generalizable to various pre-trained language models and represents an effective and practical solution for text classification in low-resource scenarios.
- Text classification /
- Multi-view fusion /
- Text clustering /
- Pre-trained language models /
- Intermediate training

HTML全文

图 1 基于HMVSF的文本分类模型框架图(以BERT为基准模型)

下载: 全尺寸图片幻灯片

图 2 路线A下不同标注预算时各模型分类准确率

下载: 全尺寸图片幻灯片

图 3 路线B下不同标注预算时各模型分类准确率

下载: 全尺寸图片幻灯片

图 4 不同簇数对CCS-BERT_IT:CLUST分类性能影响

下载: 全尺寸图片幻灯片

图 5 权重系数$ \partial $对CCS-LDA-BERT_IT:CLUST分类性能影响

下载: 全尺寸图片幻灯片

表 1 数据集和聚类一致样本信息统计

数据集	训练集	聚类一致样本集合	测试集	类别数
DBpedia	15K	12K	70K	14
AGNews	15K	11K	7.6K	4
ISEAR	5.4K	5.2K	1.5K	7
SMS Spam	3.9K	3.8K	1.1K	2
Subjectivity	7K	6.9K	2K	2
Polarity	7.5K	7.4K	2.1K	2

下载: 导出CSV

表 2 路线A下各模型的分类结果(标注预算为64)

数据集/模型	BERT	BERT_IT:CLUST	CCS-BERT_IT:CLUST	精度增益	误差减少
DBpedia	31.28	68.99	74.95	8%	19%
AGNews	66.44	78.77	82.43	5%	17%
ISEAR	18.33	26.87	29.44	9%	10%
SMS Spam	88.34	96.62	98.45	2%	54%
Subjectivity	86.93	87.97	90.61	3%	22%
Polarity	65.10	66.10	70.00	6%	12%

下载: 导出CSV

表 3 路线B下各模型的分类结果(标注预算为64)

数据集/模型	BERT_IT:MLM	BERT_IT:MLM+CLUST	CCS-BERT_IT:MLM+CLUST	精度增益	误差减少
DBpedia	55.87	75.86	79.82	5%	5%
AGNews	79.44	84.43	86.07	2%	11%
ISEAR	20.13	31.80	32.96	4%	2%
SMS Spam	86.53	98.60	99.10	1%	36%
Subjectivity	91.58	91.15	91.57	0%	5%
Polarity	70.45	71.12	74.93	5%	13%

下载: 导出CSV

表 4 聚类一致样本的NMI对比

数据集	测试集NMI		聚类一致样本NMI
	sIB	K-means	sIB	K-means
DBpedia	0.695	0.613	0.692	0.632
AGNews	0.343	0.169	0.327	0.172
ISEAR	0.108	0.042	0.111	0.075
SMS Spam	0.085	0.079	0.097	0.111
Subjectivity	0.101	0.055	0.096	0.063
Polarity	0.018	0.013	0.017	0.011

下载: 导出CSV

表 5 簇内平均欧氏距离对比

数据集	训练集	难例样本	聚类一致样本
DBpedia	–	5.88	5.09
AGNews	–	4.66	4.37
ISEAR	2.98	–	2.94
SMS Spam	2.90	–	2.81
Subjectivity	3.41	–	3.39
Polarity	3.21	–	3.20

下载: 导出CSV

表 6 本文模型与其他变体模型的分类性能对比(标注预算为64)

模型/数据集	DBpedia	AGNews	ISEAR	SMS Spam	Subjectivity	Polarity
RoBERTa	32.58	65.93	15.19	86.10	90.55	54.66
DistilBERT	42.83	64.86	16.92	86.10	89.30	66.53
ALBERT	50.90	53.18	23.92	94.89	82.55	57.43
DeBERTa	27.40	48.37	14.12	93.00	78.25	57.38
DisB-SIB-KmeansFT	73.02	78.86	27.29	96.95	91.60	63.26
RoB-SIB-KmeansFT	74.42	82.64	32.62	98.20	92.00	75.82
Be-SIB-KmeansFT	77.41	84.64	32.88	98.74	93.80	69.82
CCS-BERT_IT:CLUST	74.95	82.43	29.44	98.45	90.61	70.00
CCS-BERT_IT:MLM+CLUST	79.82	86.07	32.96	99.10	91.57	74.93
CCS-LDA-BERT_IT:CLUST	75.28	86.18	30.71	98.48	91.10	71.50
CCS-LDA-BERT_IT:MLM+CLUST	80.32	86.41	33.44	99.28	91.58	74.96

下载: 导出CSV

表 7 路线A下不同结构变体模型的分类性能对比(标注预算为64)

数据集/模型	BERT	CCS-BERT_IT:CLUST	LDA-BERT_IT:CLUST	CCS-LDA-BERT_IT:CLUST
DBpedia	31.28	74.95	43.68	75.28
AGNews	66.44	82.43	68.72	86.18
ISEAR	18.33	29.44	22.05	30.71
SMSSpam	88.34	98.45	98.56	98.48
Subjectivity	86.93	90.61	90.50	91.10
Polarity	65.10	70.00	69.39	71.50

下载: 导出CSV

参考文献(18)

[1]	WANG Kunze, DING Yihao, and HAN S C. Graph neural networks for text classification: A survey[J]. Artificial Intelligence Review, 2024, 57(8): 190. doi: 10.1007/s10462-024-10808-0.
[2]	DEVLIN J, CHANG M W, LEE K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding[C]. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, Minnesota, 2019: 4171–4186. doi: 10.18653/v1/N19-1423.
[3]	NI Shiwen and KAO H Y. KPT++: Refined knowledgeable prompt tuning for few-shot text classification[J]. Knowledge-Based Systems, 2023, 274: 110647. doi: 10.1016/j.knosys.2023.110647.
[4]	WANG Yimeng, YANG Zhiyao, and CHE Xiangjiu. A hierarchical mixture-of-experts framework for few labeled node classification[J]. Neural Networks, 2025, 188: 107285. doi: 10.1016/j.neunet.2025.107285.
[5]	CUI Yiming, CHE Wanxiang, LIU Ting, et al. Pre-training with whole word masking for Chinese BERT[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021, 29: 3504–3514. doi: 10.1109/TASLP.2021.3124365.
[6]	SHNARCH E, GERA A, HALFON A, et al. Cluster & tune: Boost cold start performance in text classification[C]. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, Dublin, Ireland, 2022: 7639–7653. doi: 10.18653/v1/2022.acl-long.526.
[7]	YU Tao, SONG Rui, PINTO S, et al. GEML: A graph-enhanced pre-trained language model framework for text classification via mutual learning[J]. Applied Intelligence, 2024, 54(23): 12215–12229. doi: 10.1007/s10489-024-05831-1.
[8]	CHEN Junfan, ZHANG Richong, ZHENG Yaowei, et al. DualCL: Principled supervised contrastive learning as mutual information maximization for text classification[C]. Proceedings of the ACM Web Conference 2024, Singapore, Singapore, 2024: 4362–4371. doi: 10.1145/3589334.3645668.
[9]	ZHOU Nai, YAO Nianmin, HU Nannan, et al. CDGAN-BERT: Adversarial constraint and diversity discriminator for semi-supervised text classification[J]. Knowledge-Based Systems, 2024, 284: 111291. doi: 10.1016/j.knosys.2023.111291.
[10]	ZOU Haochen and WANG Zitao. A semi-supervised short text sentiment classification method based on improved bert model from unlabelled data[J]. Journal of Big Data, 2023, 10(1): 35. doi: 10.1186/s40537-023-00710-x.
[11]	LI Shujie, YUAN Guanghu, YANG Min, et al. Improving semi-supervised text classification with dual meta-learning[J]. ACM Transactions on Information Systems, 2024, 42(4): 109. doi: 10.1145/3648612.
[12]	LIAO Wenxiong, LIU Zhengliang, DAI Haixing, et al. Mask-guided BERT for few-shot text classification[J]. Neurocomputing, 2024, 610: 128576. doi: 10.1016/j.neucom.2024.128576.
[13]	HOU Zejiang, SALAZAR J, and POLOVETS G. Meta-learning the difference: Preparing large language models for efficient adaptation[J]. Transactions of the Association for Computational Linguistics, 2022, 10: 1249–1265. doi: 10.1162/tacl_a_00517.
[14]	HE Kai, MAO Rui, HUANG Yucheng, et al. Template-free prompting for few-shot named entity recognition via semantic-enhanced contrastive learning[J]. IEEE transactions on Neural Networks and Learning Systems, 2024, 35(12): 18357–18369. doi: 10.1109/TNNLS.2023.3314807.
[15]	李俊辉, 侯兴松. 基于伪监督注意力短期记忆与多尺度去伪影网络的图像分块压缩感知[J]. 电子与信息学报, 2024, 46(2): 472–480. doi: 10.11999/JEIT231069. LI Junhui and HOU Xingsong. Pseudo supervised attention short-term memory and multi-scale deartifacting network based on image block compressed sensing[J]. Journal of Electronics & Information Technology, 2024, 46(2): 472–480. doi: 10.11999/JEIT231069.
[16]	CHU Yonghe, CAO Heling, DIAO Yufeng, et al. Refined SBERT: Representing sentence BERT in manifold space[J]. Neurocomputing, 2023, 555: 126453. doi: 10.1016/j.neucom.2023.126453.
[17]	ALSUHAIBANI A, ZOGAN H, RAZZAK I, et al. IDoFew: Intermediate training using dual-clustering in language models for few labels text classification[C]. Proceedings of the 17th ACM International Conference on Web Search and Data Mining, Merida, Mexico, 2024: 18–27. doi: 10.1145/3616855.3635849.
[18]	杨春玲, 梁梓文. 静态与动态域先验增强的两阶段视频压缩感知重构网络[J]. 电子与信息学报, 2024, 46(11): 4247–4258. doi: 10.11999/JEIT240295. YANG Chunling and LIANG Ziwen. Static and dynamic-domain prior enhancement two-stage video compressed sensing reconstruction network[J]. Journal of Electronics & Information Technology, 2024, 46(11): 4247–4258. doi: 10.11999/JEIT240295.