Advanced Search
Turn off MathJax
Article Contents
SUN Chao, DU Bo. Vision-Language Pretraining Enhanced Audio-Visual Multimodal Representation and Fusion Framework[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260793
Citation: SUN Chao, DU Bo. Vision-Language Pretraining Enhanced Audio-Visual Multimodal Representation and Fusion Framework[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260793

Vision-Language Pretraining Enhanced Audio-Visual Multimodal Representation and Fusion Framework

doi: 10.11999/JEIT260793 cstr: 32379.14.JEIT260793
Funds:  China Postdoctoral Science Foundation - CCTEG Joint Support Program (2026T001ZGMK), NSFC Young Students Basic Research Program (Undergraduate) (626B1007), National Key Research and Development Program of China (2023YFC2705700)
  • Accepted Date: 2026-11-05
  • Rev Recd Date: 2026-11-05
  • Available Online: 2026-11-20
  •   Objective  Audio-visual event localization (AVEL) aims to identify temporal segments where auditory and visual cues are semantically synchronized and assign corresponding event categories. Although existing methods have achieved promising results through dual-stream feature extraction and fusion, they remain vulnerable to modality heterogeneity, background noise, and semantic misalignment in real-world scenarios. To address these challenges, this paper proposes a Vision-Language Pretraining enhanced Dual-Modal Synesthesia Framework (DMSF) for robust audio-visual event localization.  Methods  DMSF initializes the audio and visual encoders using knowledge transferred from vision-language pretrained models and introduces lightweight Adapter modules for task-specific adaptation while preserving the pretrained semantic representations. To mitigate intra-modal noise, a Single-modal Noise Suppression (SNS) module is designed, which leverages state-space modeling and gated self-attention to enhance temporal dependency modeling within each modality. For cross-modal interaction, a Cross-modal Information Collaboration (CIC) module based on multi-head cross-attention is introduced to establish fine-grained audio-visual alignment. Furthermore, a visual auxiliary branch is employed to preserve relatively clean visual semantics, and a mutual learning strategy is adopted to iteratively reduce fusion noise and improve representation consistency.  Results and Discussions  Experiments conducted on the AVE dataset demonstrate that DMSF achieves superior performance under both fully supervised and weakly supervised settings. In particular, the proposed method attains accuracies of 81.7% and 76.8% under fully supervised and weakly supervised settings, respectively. The performance improvement under weak supervision scenarios indicates that the mutual-learning strategy provides effective cross-branch constraints when fine-grained temporal annotations are unavailable. Ablation studies further validate the effectiveness of each component, including SNS, CIC, and the mutual learning mechanism. Visualization results show that DMSF produces more concentrated and semantically consistent attention distributions compared to baseline methods, especially in noisy and complex environments.  Conclusions  This paper presents a unified audio-visual event localization framework that integrates vision-language pretraining, intra-modal noise suppression, cross-modal collaboration, and multi-branch mutual learning. The proposed DMSF effectively enhances multimodal representation quality and mitigates noise propagation during audio-visual interaction. Extensive experimental results confirm its superiority and robustness. Future work will focus on improving instance-level semantic alignment and enhancing model interpretability in open-world multimodal scenarios.
  • loading
  • [1]
    李玉鑑. 认知相对论—通向强人工智能之路[J]. 电子与信息学报, 2024, 46(2): 408–427. doi: 10.11999/JEIT230749.

    LI Yujian. Theory of cognitive relativity — the road to strong artificial intelligence[J]. Journal of Electronics & Information Technology, 2024, 46(2): 408–427. doi: 10.11999/JEIT230749.
    [2]
    李哲, 王可, 王彪, 等. 人机融合智能决策: 概念、框架与应用[J]. 电子与信息学报, 2025, 47(10): 3439–3464. doi: 10.11999/JEIT250260.

    LI Zhe, WANG Ke, WANG Biao, et al. Human-machine fusion intelligent decision-making: Concepts, framework, and applications[J]. Journal of Electronics & Information Technology, 2025, 47(10): 3439–3464. doi: 10.11999/JEIT250260.
    [3]
    李学龙. 多模态认知计算[J]. 中国科学: 信息科学, 2023, 53(1): 1–32. doi: 10.1360/SSI-2022-0226.

    LI Xuelong. Multi-modal cognitive computing[J]. Scientia Sinica (Informationis), 2023, 53(1): 1–32. doi: 10.1360/SSI-2022-0226.
    [4]
    孙强, 陈远. 多层次时空特征自适应集成与特有-共享特征融合的双模态情感识别[J]. 电子与信息学报, 2024, 46(2): 574–587. doi: 10.11999/JEIT231110.

    SUN Qiang and CHEN Yuan. Bimodal emotion recognition with adaptive integration of multi-level spatial-temporal features and specific-shared feature fusion[J]. Journal of Electronics & Information Technology, 2024, 46(2): 574–587. doi: 10.11999/JEIT231110.
    [5]
    孙立峰, 宋新航, 蒋树强, 等. 多模态协同感知与融合技术专题前言[J]. 软件学报, 2024, 35(5): 2099–2100. doi: 10.13328/j.cnki.jos.007030.

    SUN Lifeng, SONG Xinhang, JIANG Shuqiang, et al. Special topic on multimodal collaborative perception and fusion preface[J]. Journal of Software, 2024, 35(5): 2099–2100. doi: 10.13328/j.cnki.jos.007030.
    [6]
    张北辰, 李亮, 查正军, 等. 基于跨模态对比学习的视觉问答主动学习方法[J]. 计算机学报, 2022, 45(8): 1730–1745. doi: 10.11897/SP.J.1016.2022.0173.

    ZHANG Beichen, LI Liang, ZHA Zhengjun, et al. Contrastive cross-modal representation learning based active learning for visual question answer[J]. Chinese Journal of Computers, 2022, 45(8): 1730–1745. doi: 10.11897/SP.J.1016.2022.0173.
    [7]
    赵朝阳, 朱贵波, 王金桥. ChatGPT给语言大模型带来的启示和多模态大模型新的发展思路[J]. 数据分析与知识发现, 2023, 7(3): 26–35. doi: 10.11925/infotech.2096-3467.2023.0216.

    ZHAO Chaoyang, ZHU Guibo, and WANG Jinqiao. The inspiration brought by ChatGPT to LLM and the new development ideas of multi-modal large model[J]. Data Analysis and Knowledge Discovery, 2023, 7(3): 26–35. doi: 10.11925/infotech.2096-3467.2023.0216.
    [8]
    彭敏轩, 梁艳. 基于双流门控视听融合的多模态语音增强[J]. 计算机系统应用, 2025, 34(11): 127–138. doi: 10.15888/j.cnki.csa.009996.

    PENG Minxuan and LIANG Yan. Multi-modal speech enhancement based on dual-stream gated audio-visual fusion[J]. Computer Systems and Applications, 2025, 34(11): 127–138. doi: 10.15888/j.cnki.csa.009996.
    [9]
    章佳妍, 续欣莹, 谢珺, 等. 融合关键区域增强与双向语义引导的视听定位[J]. 计算机工程与应用, 2026, 62(3): 207–217. doi: 10.3778/j.issn.1002–8331.2411-0435.

    ZHANG Jiayan, XU Xinying, XIE Jun, et al. Audio-visual localization with key region enhancement and bidirectional semantic guidance[J]. Computer Engineering and Applications, 2026, 62(3): 207–217. doi: 10.3778/j.issn.1002-8331.2411-0435. doi: 10.3778/j.issn.1002–8331.2411-0435.
    [10]
    LIU Chen, YANG Liying, LI Peike, et al. Dynamic derivation and elimination: Audio visual segmentation with enhanced audio semantics[C]. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, USA, 2025: 3131–3141. doi: 10.1109/CVPR52734.2025.00298.
    [11]
    VILAÇA L, YU Yi, and VIANA P. A survey of recent advances and challenges in deep audio-visual correlation learning[J]. ACM Computing Surveys, 2025, 57(12): 299. doi: 10.1145/3696445.
    [12]
    YE Qilang, YU Zitong, SHAO Rui, et al. CAT+: Investigating and enhancing audio-visual understanding in large language models[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025, 47(10): 8674–8690. doi: 10.1109/TPAMI.2025.3582389.
    [13]
    BANDARA T R, GOSTAR A K, ABEYSINGHE P, et al. Occlusion handling for intelligent vehicles using audio-visual multi-object tracking[J]. IEEE Transactions on Intelligent Transportation Systems, 2026, 27(9): 11578–11593. doi: 10.1109/TITS.2026.3697405.
    [14]
    TIAN Yapeng, SHI Jing, LI Bochen, et al. Audio-visual event localization in unconstrained videos[C]. 2018 15th European Conference on Computer Vision (ECCV), Munich, Germany, 2018: 252–268. doi: 10.1007/978-3-030-01216-8_16.
    [15]
    XU Haoming, ZENG Runhao, WU Qingyao, et al. Cross-modal relation-aware networks for audio-visual event localization[C]. 2020 28th ACM International Conference on Multimedia (ACM MM), Seattle, USA, 2020: 3893–3901. doi: 10.1145/3394171.3413581.
    [16]
    LIN Yanbo, LI Y J, and WANG Y C F. Dual-modality seq2seq network for audio-visual event localization[C]. 2019 44th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK, 2019: 2002–2006. doi: 10.1109/ICASSP.2019.8683226.
    [17]
    LIN Yanbo and WANG Y C F. Audiovisual transformer with instance attention for audio-visual event localization[C]. 2020 15th Asian Conference on Computer Vision (ACCV), Kyoto, Japan, 2021: 274–290. doi: 10.1007/978-3-030-69544-6_17.
    [18]
    ZHOU Jinxing, ZHENG Liang, ZHONG Yiran, et al. Positive sample propagation along the audio-visual event line[C]. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, USA, 2021: 8432–8440. doi: 10.1109/CVPR46437.2021.00833.
    [19]
    XIA Yan and ZHAO Zhou. Cross-modal background suppression for audio-visual event localization[C]. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, USA, 2022: 19957–19966. doi: 10.1109/CVPR52688.2022.01936.
    [20]
    XUAN Hanyu, ZHANG Zhenyu, CHEN Shuo, et al. Cross-modal attention network for temporal inconsistent audio-visual event localization[C]. 2020 34th AAAI Conference on Artificial Intelligence (AAAI-20), New York, USA, 2020: 279–286. doi: 10.1609/aaai.v34i01.5361.
    [21]
    SUN Chao, CHEN Min, CHENG Jialiang, et al. SCLAV: Supervised cross-modal contrastive learning for audio-visual coding[C]. 2023 31st ACM International Conference on Multimedia (ACM MM), Ottawa, Canada, 2023: 261–270. doi: 10.1145/3581783.3613805.
    [22]
    SUN Chao, CHEN Min, ZHU Chuanbo, et al. Listen with seeing: Cross-modal contrastive learning for audio-visual event localization[J]. IEEE Transactions on Multimedia, 2025, 27: 2650–2665. doi: 10.1109/TMM.2025.3535359.
    [23]
    YU Jiashuo, CHENG Ying, ZHAO Ruiwei, et al. MM-pyramid: Multimodal pyramid attentional network for audio-visual event localization and video parsing[C]. 2022 30th ACM International Conference on Multimedia (ACM MM), Lisbon, Portugal, 2022: 6241–6249. doi: 10.1145/3503161.3547869.
    [24]
    HE Xiang, LIU Xiangxi, LI Yang, et al. CACE-Net: Co-guidance attention and contrastive enhancement for effective audio-visual event localization[C]. 2024 32nd ACM International Conference on Multimedia (ACM MM), Melbourne, Australia, 2024: 985–993. doi: 10.1145/3664647.3681503.
    [25]
    GUZHOV A, RAUE F, HEES J, et al. Audioclip: Extending clip to image, text and audio[C]. 2022 47th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore, Singapore, 2022: 976–980. doi: 10.1109/ICASSP43922.2022.9747631.
    [26]
    LIN Yanbo, SUNG Y L, LEI Jie, et al. Vision transformers are parameter-efficient audio-visual learners[C]. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, Canada, 2023: 2299–2309. doi: 10.1109/CVPR52729.2023.00228.
    [27]
    WANG Kai, TIAN Yapeng, and DIMITRIOS H. Towards efficient audio-visual learners via empowering pre-trained vision transformers with cross-modal adaptation[C]. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, USA, 2024: 1837–1846. doi: 10.1109/CVPRW63382.2024.00190.
    [28]
    CHENG Ying, LI Yang, HE Junjie, et al. Mixture of experts for audio-visual learning[C]. Proceedings of the 38th International Conference on Neural Information Processing Systems, Vancouver, Canada, 2024: 7. doi: 10.52202/079017-0007.
    [29]
    HUANG Longzhuo, LI Liang, FU Xueyang, et al. Efficient prompt-based multimodal interaction for audio-visual event localization[C]. Proceedings of the 2025 International Conference on Multimedia Retrieval, Chicago, USA, 2025: 506–515. doi: 10.1145/3731715.3733320.
  • 加载中

Catalog

    通讯作者: 陈斌, bchen63@163.com
    • 1. 

      沈阳化工大学材料科学与工程学院 沈阳 110142

    1. 本站搜索
    2. 百度学术搜索
    3. 万方数据库搜索
    4. CNKI搜索

    Figures(7)  / Tables(5)

    Article Metrics

    Article views (21) PDF downloads(0) Cited by()
    Proportional views
    Related

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return