A Heterogeneous Multi-View Semantic Fusion Training Method for Text Classification in Low-Resource Scenarios

XU Sen; DENG Jubiao; XU Xiufang; YAO Shanliang; BIAN Xuesheng; BEN Xianye

doi:10.11999/JEIT260365

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2026 >

XU Sen, DENG Jubiao, XU Xiufang, YAO Shanliang, BIAN Xuesheng, BEN Xianye. A Heterogeneous Multi-View Semantic Fusion Training Method for Text Classification in Low-Resource Scenarios[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260365

Citation:

XU Sen, DENG Jubiao, XU Xiufang, YAO Shanliang, BIAN Xuesheng, BEN Xianye. A Heterogeneous Multi-View Semantic Fusion Training Method for Text Classification in Low-Resource Scenarios[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260365

Citation:

XU Sen, DENG Jubiao, XU Xiufang, YAO Shanliang, BIAN Xuesheng, BEN Xianye. A Heterogeneous Multi-View Semantic Fusion Training Method for Text Classification in Low-Resource Scenarios[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260365

PDF( 3718 KB)

A Heterogeneous Multi-View Semantic Fusion Training Method for Text Classification in Low-Resource Scenarios

doi: 10.11999/JEIT260365 cstr: 32379.14.JEIT260365

XU Sen^{1, 2, 3
,
,},
DENG Jubiao^{1, 2
,},
XU Xiufang^{1, 2},
YAO Shanliang^{1, 2},
BIAN Xuesheng^{1, 2},
BEN Xianye^{3, 4}

1.
School of Information Engineering, Yancheng Institute of Technology, Yancheng 224051, China
2.
Jiangsu Provincial Engineering Center for Offshore Wind Power Multi-Modal Perception and Intelligent Control, Yancheng 224051, China
3.
College of Information and Communication Engineering, Harbin Engineering University, Harbin 150001, China
4.
School of Information Science and Engineering, Shandong University, Qingdao 266237, China

Funds: The National Natural Science Foundation of China (62676350, 52602749, 62301473, 62322111), Jiangsu University Qing Lan Project, Yancheng Industrial Innovation Technology Support (Industrial) Special Program(YCBG2025201), The Natural Science Foundation of Heilongjiang Province (ZD2025F004)

Received Date: 2026-03-30
Accepted Date: 2026-08-17
Rev Recd Date: 2026-08-17

Available Online: 2026-08-26

Abstract

Abstract

Objective The increasing demand for text classification in specialized domains, such as finance, healthcare, and social media, is often hindered by the scarcity of labeled data. Low-resource or data-scarce scenarios significantly limit the effectiveness of conventional supervised learning and pre-trained language models, such as BERT, due to insufficient task-specific supervision. Existing methods that rely on single-view weak supervision or clustering often generate noisy pseudo-labels and fail to fully exploit the discriminative potential of hard-to-classify samples. Therefore, it is essential to develop a robust training method that leverages unlabeled data, mitigates noise sensitivity, and enhances semantic representation in low-label conditions. The primary objective of this research is to propose a text classification training framework specifically designed for data-scarce scenarios. Methods To address these challenges, a Heterogeneous Multi-View Semantic Fusion (HMVSF) training framework is introduced. HMVSF consists of three main stages: (1) Dual Clustering Consistency Screening (DCCS): Text samples are first represented in the word-frequency view using TF-IDF vectors. Two heterogeneous clustering algorithms, sIB and an improved K-means initialized via a genetic algorithm, are applied to the same feature space. Samples that are consistently assigned to intersecting clusters are selected as high-confidence pseudo-labeled instances. These samples are subsequently used for a lightweight supervised update of the shared encoder, ensuring reliable initialization for subsequent training. (2) Latent Dirichlet Allocation (LDA) Guided Hard-Sample Retraining: The remaining hard examples, which are not selected in the first stage, typically exhibit low discriminability in the word-frequency view but carry global semantic and topic-level information. LDA is employed to model the topic distribution of these samples. The optimal number of topics is adaptively determined by a combination of semantic coherence and statistical fitting criteria. Hard and soft pseudo-labels are generated based on the document-topic distribution, and the model undergoes targeted retraining using a combined cross-entropy and symmetric KL divergence loss to integrate topic-level semantic cues. (3) Downstream Fine-Tuning: Finally, the encoder is fine-tuned on a small labeled set by replacing temporary heads with a task-specific classification head. Standard cross-entropy is employed as the objective. The two-stage intermediate training is independent of the pre-training initialization, allowing HMVSF to be applied to both general-purpose BERT and masked language model (MLM)-based initializations, making it widely applicable. Results and Discussions The effectiveness of HMVSF is evaluated on six publicly available datasets, including DBpedia, AGNews, ISEAR, SMS Spam, Subjectivity, and Polarity, which cover both topic-based and non-topic-based text classification tasks. Two initialization routes are tested: Route A (BERT/BERT_IT:CLUST) and Route B (BERT_IT:MLM/BERT_IT:MLM+CLUST). Under extremely low label budgets (e.g., 64 labeled samples), CCS-BERT_IT:CLUST and CCS-BERT_IT:MLM+CLUST outperform baseline models and single-stage variants across datasets (Figs.2–3, Tables 2–3). For instance, in Route A, CCS-BERT_IT:CLUST achieves up to 9% accuracy improvement on DBpedia and ISEAR compared with BERT_IT:CLUST, while error rates decrease by 10–54% (Table 2). Normalized Mutual Information (NMI) and intra-cluster Euclidean distance analyses confirm that the CCS selection produces stable and compact clusters, ensuring reliable pseudo-labels (Tables 4–5). Ablation experiments show that the two-stage design combining CCS and LDA is superior to single-stage variants, demonstrating complementary advantages: CCS provides high-confidence core samples, and LDA captures latent semantic structures of hard examples (Table 7). Parameter sensitivity analysis reveals that the model is robust under a wide range of cluster numbers and LDA weight coefficients, achieving stable performance when the LDA weighting coefficient is set to 0.3 (Figs.4–5). HMVSF also maintains competitive performance as labeled data increase. Computational experiments on an NVIDIA A40 GPU demonstrate that the two-stage intermediate training introduces acceptable additional overhead while enhancing representation quality and overall accuracy. Conclusions This study presents a novel heterogeneous multi-view semantic fusion training method (HMVSF) for data-scarce text classification. By integrating cluster-consistent sample selection in the word-frequency view and LDA-guided hard-sample retraining, HMVSF effectively leverages unlabeled data, mitigates pseudo-label noise, and enriches semantic representations. Extensive experiments across multiple datasets confirm that HMVSF significantly improves classification accuracy under low-label conditions while remaining robust to pre-training initialization. The framework is generalizable to various pre-trained language models and represents an effective and practical solution for text classification in low-resource scenarios.
- Text classification,
- Multi-view fusion,
- Text clustering,
- Pre-trained language models,
- Intermediate training

FullText(HTML)

References(18)

References

[1]	WANG Kunze, DING Yihao, and HAN S C. Graph neural networks for text classification: A survey[J]. Artificial Intelligence Review, 2024, 57(8): 190. doi: 10.1007/s10462-024-10808-0.
[2]	DEVLIN J, CHANG M W, LEE K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding[C]. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, Minnesota, 2019: 4171–4186. doi: 10.18653/v1/N19-1423.
[3]	NI Shiwen and KAO H Y. KPT++: Refined knowledgeable prompt tuning for few-shot text classification[J]. Knowledge-Based Systems, 2023, 274: 110647. doi: 10.1016/j.knosys.2023.110647.
[4]	WANG Yimeng, YANG Zhiyao, and CHE Xiangjiu. A hierarchical mixture-of-experts framework for few labeled node classification[J]. Neural Networks, 2025, 188: 107285. doi: 10.1016/j.neunet.2025.107285.
[5]	CUI Yiming, CHE Wanxiang, LIU Ting, et al. Pre-training with whole word masking for Chinese BERT[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021, 29: 3504–3514. doi: 10.1109/TASLP.2021.3124365.
[6]	SHNARCH E, GERA A, HALFON A, et al. Cluster & tune: Boost cold start performance in text classification[C]. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, Dublin, Ireland, 2022: 7639–7653. doi: 10.18653/v1/2022.acl-long.526.
[7]	YU Tao, SONG Rui, PINTO S, et al. GEML: A graph-enhanced pre-trained language model framework for text classification via mutual learning[J]. Applied Intelligence, 2024, 54(23): 12215–12229. doi: 10.1007/s10489-024-05831-1.
[8]	CHEN Junfan, ZHANG Richong, ZHENG Yaowei, et al. DualCL: Principled supervised contrastive learning as mutual information maximization for text classification[C]. Proceedings of the ACM Web Conference 2024, Singapore, Singapore, 2024: 4362–4371. doi: 10.1145/3589334.3645668.
[9]	ZHOU Nai, YAO Nianmin, HU Nannan, et al. CDGAN-BERT: Adversarial constraint and diversity discriminator for semi-supervised text classification[J]. Knowledge-Based Systems, 2024, 284: 111291. doi: 10.1016/j.knosys.2023.111291.
[10]	ZOU Haochen and WANG Zitao. A semi-supervised short text sentiment classification method based on improved bert model from unlabelled data[J]. Journal of Big Data, 2023, 10(1): 35. doi: 10.1186/s40537-023-00710-x.
[11]	LI Shujie, YUAN Guanghu, YANG Min, et al. Improving semi-supervised text classification with dual meta-learning[J]. ACM Transactions on Information Systems, 2024, 42(4): 109. doi: 10.1145/3648612.
[12]	LIAO Wenxiong, LIU Zhengliang, DAI Haixing, et al. Mask-guided BERT for few-shot text classification[J]. Neurocomputing, 2024, 610: 128576. doi: 10.1016/j.neucom.2024.128576.
[13]	HOU Zejiang, SALAZAR J, and POLOVETS G. Meta-learning the difference: Preparing large language models for efficient adaptation[J]. Transactions of the Association for Computational Linguistics, 2022, 10: 1249–1265. doi: 10.1162/tacl_a_00517.
[14]	HE Kai, MAO Rui, HUANG Yucheng, et al. Template-free prompting for few-shot named entity recognition via semantic-enhanced contrastive learning[J]. IEEE transactions on Neural Networks and Learning Systems, 2024, 35(12): 18357–18369. doi: 10.1109/TNNLS.2023.3314807.
[15]	李俊辉, 侯兴松. 基于伪监督注意力短期记忆与多尺度去伪影网络的图像分块压缩感知[J]. 电子与信息学报, 2024, 46(2): 472–480. doi: 10.11999/JEIT231069. LI Junhui and HOU Xingsong. Pseudo supervised attention short-term memory and multi-scale deartifacting network based on image block compressed sensing[J]. Journal of Electronics & Information Technology, 2024, 46(2): 472–480. doi: 10.11999/JEIT231069.
[16]	CHU Yonghe, CAO Heling, DIAO Yufeng, et al. Refined SBERT: Representing sentence BERT in manifold space[J]. Neurocomputing, 2023, 555: 126453. doi: 10.1016/j.neucom.2023.126453.
[17]	ALSUHAIBANI A, ZOGAN H, RAZZAK I, et al. IDoFew: Intermediate training using dual-clustering in language models for few labels text classification[C]. Proceedings of the 17th ACM International Conference on Web Search and Data Mining, Merida, Mexico, 2024: 18–27. doi: 10.1145/3616855.3635849.
[18]	杨春玲, 梁梓文. 静态与动态域先验增强的两阶段视频压缩感知重构网络[J]. 电子与信息学报, 2024, 46(11): 4247–4258. doi: 10.11999/JEIT240295. YANG Chunling and LIANG Ziwen. Static and dynamic-domain prior enhancement two-stage video compressed sensing reconstruction network[J]. Journal of Electronics & Information Technology, 2024, 46(11): 4247–4258. doi: 10.11999/JEIT240295.