Research on Multimodal Sentiment Analysis Method Based onAdversarial Optimization and Triple Soft Contrastive Learning

KANG Shouqiang; ZHANG Bohao; XIE Jinbao

doi:10.11999/JEIT260255

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2025 >

KANG Shouqiang, ZHANG Bohao, XIE Jinbao. Research on Multimodal Sentiment Analysis Method Based onAdversarial Optimization and Triple Soft Contrastive Learning[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260255

Citation:

KANG Shouqiang, ZHANG Bohao, XIE Jinbao. Research on Multimodal Sentiment Analysis Method Based onAdversarial Optimization and Triple Soft Contrastive Learning[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260255

Citation:

PDF( 3048 KB)

Research on Multimodal Sentiment Analysis Method Based onAdversarial Optimization and Triple Soft Contrastive Learning

doi: 10.11999/JEIT260255 cstr: 32379.14.JEIT260255

1.
Department of Harbin University of Science and Technology Heilongjiang Provincial Key Laboratory of Pattern Recognition and Information Perception, Heilongjiang 150080, China
2.
College of Physics and Electronic Engineering, Hainan Normal University, Hainan 571158, China

Received Date: 2026-03-09
Accepted Date: 2026-09-13
Rev Recd Date: 2026-09-12

Available Online: 2026-09-18

Abstract

Abstract

Objective Multimodal sentiment analysis is an important research topic in the field of human-computer interaction because it integrates information from multiple modalities, such as text, audio, and video, to understand human emotions. However, existing methods still have several limitations. First, many approaches neglect the temporal dependencies among multimodal features, resulting in insufficient utilization of temporal information. Second, conventional multimodal fusion strategies usually rely on simple feature concatenation, which limits cross-modal interaction and may introduce redundant features. Moreover, sentiment expressions are continuous and ambiguous, whereas traditional hard contrastive learning strategies generally model sample relationships in a binary manner, making it difficult to capture subtle semantic differences among samples. To address these problems, this paper proposes a multimodal sentiment analysis method based on adversarial optimization and triple soft contrastive learning. The proposed framework enhances multimodal representation learning by modeling temporal dependencies and introducing a soft contrastive learning mechanism that assigns continuous similarity weights between samples. Meanwhile, adversarial optimization is incorporated to improve the robustness and discriminative capability of the learned representations. Methods The proposed method constructs a triple soft contrastive learning framework with adversarial optimization. The overall architecture consists of three components: multimodal feature extraction, triple soft contrastive representation learning, and adversarial optimization. First, modality-specific encoders are employed to extract representations from textual, acoustic, and visual inputs. The extracted features are mapped into a shared representation space to facilitate multimodal interaction. To capture temporal dependencies within multimodal data, a temporal contrastive learning strategy is introduced to model the relationships between sequential features and preserve temporally consistent representations. Second, a soft contrastive learning mechanism is adopted to overcome the limitations of conventional hard contrastive learning. Instead of using binary labels to distinguish positive and negative samples, the proposed method assigns continuous similarity weights according to the sentiment similarity between samples, enabling the model to better capture gradual sentiment transitions and alleviate the influence of ambiguous sentiment boundaries. Third, an adversarial optimization strategy is introduced to further improve the robustness of representation learning. By constructing a dynamic adversarial training process between representation learning and adversarial perturbation, the model learns more discriminative and stable multimodal representations. The overall training objective combines the soft contrastive loss, temporal contrastive loss, and adversarial loss to achieve effective multimodal representation learning. The overall framework of the proposed model is illustrated in (Fig. 2). Results and Discussions The effectiveness of the proposed method is evaluated through extensive experiments on widely used multimodal sentiment analysis datasets. The experimental results demonstrate that the proposed adversarial optimization-based triple soft contrastive learning framework improves the performance of multimodal sentiment analysis. Compared with representative baseline models, the proposed method achieves competitive performance in terms of classification accuracy, F1-score, and regression metrics. The comparisons with existing multimodal methods are summarized in (Tables 5 and 6). The results indicate that soft contrastive learning enables the model to capture fine-grained sentiment relationships between samples, while adversarial optimization improves the robustness of representations against noisy inputs. Furthermore, ablation studies verify the contribution of each component to performance improvement. In particular, the soft contrastive learning mechanism effectively alleviates the limitations of hard boundary assumptions in traditional contrastive learning, while temporal contrastive learning preserves temporal sentiment information. The adversarial optimization module further enhances the discriminative capability of learned representations. These results demonstrate the effectiveness of combining adversarial optimization with contrastive representation learning for multimodal sentiment analysis. Conclusions This paper proposes a multimodal sentiment analysis method based on adversarial optimization and triple soft contrastive learning. The proposed approach introduces a soft contrastive learning strategy to model continuous sentiment similarity between samples, while temporal contrastive learning captures temporal dependencies in multimodal data. In addition, adversarial optimization is incorporated to enhance representation robustness and improve generalization capability. Experimental results on benchmark datasets demonstrate that the proposed method achieves competitive performance and effectively improves multimodal sentiment representation learning. The proposed framework provides a potential solution for robust and fine-grained multimodal sentiment analysis.
- Multimodal Sentiment Analysis (MSA),
- Soft contrastive learning,
- Generative Adversarial Network (GAN),
- Multimodal learning,
- Feature fusion

FullText(HTML)

References(35)

References

[1]	黄辰, 刘会杰, 张龑, 等. 带全局噪声增强的多模态超图学习引导用于模态信息缺失情感分析[J]. 电子与信息学报, 2025, 47(12): 5192–5202. doi: 10.11999/JEIT250649. HUANG Chen, LIU Huijie, ZHANG Yan, et al. Multimodal hypergraph learning guidance with global noise enhancement for sentiment analysis under missing modality information[J]. Journal of Electronics & Information Technology, 2025, 47(12): 5192–5202. doi: 10.11999/JEIT250649.
[2]	张乐, 陈岩松, 张雷瀚. 大模型特征增强与多层次交叉融合的多模态情感分析方法[J]. 数据分析与知识发现, 2025, 9(8): 47–58. doi: 10.11925/infotech.2096-3467.2024.0625. ZHANG Le, CHEN Yansong, and ZHANG Leihan. A multimodal sentiment analysis method based on LLM feature enhancement and multi-level cross-fusion[J]. Data Analysis and Knowledge Discovery, 2025, 9(8): 47–58. doi: 10.11925/infotech.2096-3467.2024.0625.
[3]	陈杰, 马静, 李晓峰, 等. 基于DR-Transformer模型的多模态情感识别研究[J]. 情报科学, 2022, 40(3): 117–125. doi: 10.13833/j.issn.1007-7634.2022.03.015. CHEN Jie, MA Jing, LI Xiaofeng, et al. Multi-modal emotion recognition based on DR-Transformer model[J]. Information Science, 2022, 40(3): 117–125. doi: 10.13833/j.issn.1007-7634.2022.03.015.
[4]	林宜山, 左景, 卢树华. 基于音视频特征优化与跨模态Transformer的多模态情感分析[J]. 北京航空航天大学学报, 2026, 52(6): 2219–2228. doi: 10.13700/j.bh.1001-5965.2024.0247. LIN Yishan, ZUO Jing, and LU Shuhua. A multimodal sentiment analysis based on audio and video features optimization and cross-modal Transformer[J]. Journal of Beijing University of Aeronautics and Astronautics, 2026, 52(6): 2219–2228. doi: 10.13700/j.bh.1001-5965.2024.0247.
[5]	WU Yujin, DAOUDI M, and AMAD A. Transformer-based self-supervised multimodal representation learning for wearable emotion recognition[J]. IEEE Transactions on Affective Computing, 2024, 15(1): 157–172. doi: 10.1109/TAFFC.2023.3263907.
[6]	CAI Yujian, LI Xingguang, ZHANG Yingyu, et al. Multimodal sentiment analysis based on multi-layer feature fusion and multi-task learning[J]. Scientific Reports, 2025, 15(1): 2126. doi: 10.1038/s41598-025-85859-6.
[7]	冯广, 周垣桦, 钟婷, 等. 结合自适应特征加权与权值优化策略的多模态情感分析[J]. 计算机工程与应用, 2026, 62(6): 194–204. doi: 10.3778/j.issn.1002-8331.2501-0164. FENG Guang, ZHOU Yuanhua, ZHONG Ting, et al. Multimodal sentiment analysis combining adaptive feature weighting and weight optimization strategy[J]. Computer Engineering and Applications, 2026, 62(6): 194–204. doi: 10.3778/j.issn.1002-8331.2501-0164.
[8]	刘佳, 宋泓, 陈大鹏, 等. 非语言信息增强和对比学习的多模态情感分析模型[J]. 电子与信息学报, 2024, 46(8): 3372–3381. doi: 10.11999/JEIT231274. LIU Jia, SONG Hong, CHEN Dapeng, et al. A multimodal sentiment analysis model enhanced with non-verbal information and contrastive learning[J]. Journal of Electronics & Information Technology, 2024, 46(8): 3372–3381. doi: 10.11999/JEIT231274.
[9]	SCHULLER B, RIGOLL G, and LANG M. Speech emotion recognition: Features and classification[J]. Speech Communication, 2009, 51(10): 975–982. doi: 10.1016/j.dsp.2012.05.007.
[10]	YE Jiaxin, WEN Xincheng, WEI Yujie, et al. Temporal modeling matters: A novel temporal emotional modeling approach for speech emotion recognition[C]. The IEEE International Conference on Acoustics, Speech and Signal Processing, Rhodes Island, Greece, 2023: 1–5. doi: 10.1109/ICASSP49357.2023.10096370.
[11]	MIKOLOV T, CHEN Kai, CORRADO G, et al. Efficient estimation of word representations in vector space[C]. International Conference on Learning Representations, Scottsdale, USA, 2013: 1301–3781.
[12]	DEVLIN J, CHANG Mingwei, LEE K, et al. BERT: Pre-training of deep bidirectional Transformers for language understanding[C]. The 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, Minnesota, 2019: 4171–4186. doi: 10.18653/v1/N19-1423.
[13]	WANG Yabing, HUANG Guimin, LI Maolin, et al. Automatically constructing a fine-grained sentiment lexicon for sentiment analysis[J]. Cognitive Computation, 2023, 15(1): 254–271. doi: 10.1007/s12559-022-10043-1.
[14]	JASSIM M A, ABD D H, and OMRI M N. A survey of sentiment analysis from film critics based on machine learning, lexicon and hybridization[J]. Neural Computing and Applications, 2023, 35(13): 9437–9461. doi: 10.1007/s00521-023-08359-6.
[15]	曹银妮, 韩虎, 黄明伟, 等. 基于多视角融合表示的多模态方面级情感分析模型[J]. 数据分析与知识发现, 2025, 9(10): 54–67. doi: 10.11925/infotech.2096-3467.2024.1114. CAO Yinni, HAN Hu, HUANG Mingwei, et al. Multi-modal aspect-level sentiment analysis model with multi-view fusion representation[J]. Data Analysis and Knowledge Discovery, 2025, 9(10): 54–67. doi: 10.11925/infotech.2096-3467.2024.1114.
[16]	赵川斌, 许伟华, 林博, 等. 融合视觉的多模态通信感知一体化关键技术及原型验证[J]. 电子与信息学报, 2026, 48(2): 487–498. doi: 10.11999/JEIT250685. ZHAO Chuanbin, XU Weihua, LIN Bo, et al. Vision enabled multimodal integrated sensing and communications: Key technologies and prototype validation[J]. Journal of Electronics & Information Technology, 2026, 48(2): 487–498. doi: 10.11999/JEIT250685.
[17]	YOU Quanzeng, JIN Hailin, and LUO Jiebo. Visual sentiment analysis by attending on local image regions[C]. The 31st AAAI Conference on Artificial Intelligence, San Francisco, USA, 2017: 231–237.
[18]	LIU Yunze, FAN Qingnan, ZHANG Shanghang, et al. Contrastive multimodal fusion with TupleInfoNCE[C]. The IEEE/CVF International Conference on Computer Vision, Montreal, Canada, 2021: 734–743. doi: 10.1109/ICCV48922.2021.00079.
[19]	YANG Liu, WU Zhenjie, HONG Junkun, et al. MCL: A contrastive learning method for multimodal data fusion in violence detection[J]. IEEE Signal Processing Letters, 2023, 30: 408–412. doi: 10.1109/LSP.2022.3227818.
[20]	GRILL J B, STRUB F, ALTCHÉ F, et al. Bootstrap your own latent a new approach to self-supervised learning[C]. The 34th International Conference on Neural Information Processing Systems, Vancouver, Canada, 2020: 1786.
[21]	WANG Huiru, LI Xiuhong, REN Zenyu, et al. Multimodal sentiment analysis representations learning via contrastive learning with condense attention fusion[J]. Sensors, 2023, 23(5): 2679. doi: 10.3390/s23052679.
[22]	MAI Sijie, ZENG Ying, ZHENG Shuangjia, et al. Hybrid contrastive learning of tri-modal representation for multimodal sentiment analysis[J]. IEEE Transactions on Affective Computing, 2023, 14(3): 2276–2289. doi: 10.1109/TAFFC.2022.3172360.
[23]	QUAN Zhibang, SUN Tao, SU Mengli, et al. Multimodal sentiment analysis based on nonverbal representation optimization network and contrastive interaction learning[C]. The IEEE International Conference on Systems, Man, and Cybernetics, Prague, Czech Republic, 2022: 3086–3091. doi: 10.1109/SMC53654.2022.9945514.
[24]	YANG Jiuding, YU Yakun, NIU Di, et al. ConFEDE: Contrastive feature decomposition for multimodal sentiment analysis[C]. The 61st Annual Meeting of the Association for Computational Linguistics, Toronto, Canada, 2023: 7617–7630. doi: 10.18653/v1/2023.acl-long.421.
[25]	WANG Senzhang, YAN Hao, DU Jinlong, et al. Adversarial hard negative generation for complementary graph contrastive learning[C]. SIAM International Conference on Data Mining, Austin, USA, 2023: 163–171. doi: 10.1137/1.9781611977653.ch19.
[26]	ZADEH A, ZELLERS R, PINCUS E, et al. MOSI: Multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos[EB/OL]. https://arxiv.org/abs/1606.06259, 2016.
[27]	ZADEH A, LIANG P P, PORIA S, et al. Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph[C]. The 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia, 2018: 2236–2246. doi: 10.18653/v1/P18-1208.
[28]	ZADEH A, CHEN Minghai, PORIA S, et al. Tensor fusion network for multimodal sentiment analysis[C]. The 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark, 2017: 1103–1114. doi: 10.18653/v1/D17-1115.
[29]	LIU Zhun, SHEN Ying, LAKSHMINARASIMHAN V B, et al. Efficient low-rank multimodal fusion with modality-specific factors[C]. The 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia, 2018: 2247–2256. doi: 10.18653/v1/P18-1209.
[30]	YU Wenmeng, XU Hua, YUAN Ziqi, et al. Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis[C]. The 35th AAAI Conference on Artificial Intelligence, 2021: 10790–10797. doi: 10.1609/aaai.v35i12.17289.
[31]	HUANG Jian, JI Yanli, YANG Yang, et al. Cross-modality representation interactive learning for multimodal sentiment analysis[C]. The 31st ACM International Conference on Multimedia, Ottawa, Canada, 2023: 426–434. doi: 10.1145/3581783.3612295.
[32]	RAHMAN W, HASAN K, LEE S, et al. Integrating multimodal information in large pretrained transformers[C]. The 58th Annual Meeting of the Association for Computational Linguistics, 2020: 2359–2369. doi: 10.18653/v1/2020.acl-main.214.
[33]	HAZARIKA D, ZIMMERMANN R, and PORIA S. MISA: Modality-invariant and -specific representations for multimodal sentiment analysis[C]. The 28th ACM International Conference on Multimedia, Seattle, USA, 2020: 1122–1131.
[34]	ZHAO Wang, ZHANG Yong, HUA Qiang, et al. MSA-HCL: Multimodal sentiment analysis model with hybrid contrastive learning[J]. Mathematical Foundations of Computing, 2025, 8(3): 433–447. doi: 10.3934/mfc.2024017.
[35]	ZHUANG Yan, BAI Wei, ZHANG Yanru, et al. Multi-level contrastive learning for multimodal sentiment analysis[J]. IEEE Transactions on Multimedia, 2025, 27: 9044–9058. doi: 10.1109/TMM.2025.3613116.