| Citation: | SUN Chao, DU Bo. Vision-Language Pretraining Enhanced Audio-Visual Multimodal Representation and Fusion Framework[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260793 |
| [1] |
李玉鑑. 认知相对论—通向强人工智能之路[J]. 电子与信息学报, 2024, 46(2): 408–427. doi: 10.11999/JEIT230749.
LI Yujian. Theory of cognitive relativity — the road to strong artificial intelligence[J]. Journal of Electronics & Information Technology, 2024, 46(2): 408–427. doi: 10.11999/JEIT230749.
|
| [2] |
李哲, 王可, 王彪, 等. 人机融合智能决策: 概念、框架与应用[J]. 电子与信息学报, 2025, 47(10): 3439–3464. doi: 10.11999/JEIT250260.
LI Zhe, WANG Ke, WANG Biao, et al. Human-machine fusion intelligent decision-making: Concepts, framework, and applications[J]. Journal of Electronics & Information Technology, 2025, 47(10): 3439–3464. doi: 10.11999/JEIT250260.
|
| [3] |
李学龙. 多模态认知计算[J]. 中国科学: 信息科学, 2023, 53(1): 1–32. doi: 10.1360/SSI-2022-0226.
LI Xuelong. Multi-modal cognitive computing[J]. Scientia Sinica (Informationis), 2023, 53(1): 1–32. doi: 10.1360/SSI-2022-0226.
|
| [4] |
孙强, 陈远. 多层次时空特征自适应集成与特有-共享特征融合的双模态情感识别[J]. 电子与信息学报, 2024, 46(2): 574–587. doi: 10.11999/JEIT231110.
SUN Qiang and CHEN Yuan. Bimodal emotion recognition with adaptive integration of multi-level spatial-temporal features and specific-shared feature fusion[J]. Journal of Electronics & Information Technology, 2024, 46(2): 574–587. doi: 10.11999/JEIT231110.
|
| [5] |
孙立峰, 宋新航, 蒋树强, 等. 多模态协同感知与融合技术专题前言[J]. 软件学报, 2024, 35(5): 2099–2100. doi: 10.13328/j.cnki.jos.007030.
SUN Lifeng, SONG Xinhang, JIANG Shuqiang, et al. Special topic on multimodal collaborative perception and fusion preface[J]. Journal of Software, 2024, 35(5): 2099–2100. doi: 10.13328/j.cnki.jos.007030.
|
| [6] |
张北辰, 李亮, 查正军, 等. 基于跨模态对比学习的视觉问答主动学习方法[J]. 计算机学报, 2022, 45(8): 1730–1745. doi: 10.11897/SP.J.1016.2022.0173.
ZHANG Beichen, LI Liang, ZHA Zhengjun, et al. Contrastive cross-modal representation learning based active learning for visual question answer[J]. Chinese Journal of Computers, 2022, 45(8): 1730–1745. doi: 10.11897/SP.J.1016.2022.0173.
|
| [7] |
赵朝阳, 朱贵波, 王金桥. ChatGPT给语言大模型带来的启示和多模态大模型新的发展思路[J]. 数据分析与知识发现, 2023, 7(3): 26–35. doi: 10.11925/infotech.2096-3467.2023.0216.
ZHAO Chaoyang, ZHU Guibo, and WANG Jinqiao. The inspiration brought by ChatGPT to LLM and the new development ideas of multi-modal large model[J]. Data Analysis and Knowledge Discovery, 2023, 7(3): 26–35. doi: 10.11925/infotech.2096-3467.2023.0216.
|
| [8] |
彭敏轩, 梁艳. 基于双流门控视听融合的多模态语音增强[J]. 计算机系统应用, 2025, 34(11): 127–138. doi: 10.15888/j.cnki.csa.009996.
PENG Minxuan and LIANG Yan. Multi-modal speech enhancement based on dual-stream gated audio-visual fusion[J]. Computer Systems and Applications, 2025, 34(11): 127–138. doi: 10.15888/j.cnki.csa.009996.
|
| [9] |
章佳妍, 续欣莹, 谢珺, 等. 融合关键区域增强与双向语义引导的视听定位[J]. 计算机工程与应用, 2026, 62(3): 207–217. doi: 10.3778/j.issn.1002–8331.2411-0435.
ZHANG Jiayan, XU Xinying, XIE Jun, et al. Audio-visual localization with key region enhancement and bidirectional semantic guidance[J]. Computer Engineering and Applications, 2026, 62(3): 207–217. doi: 10.3778/j.issn.1002-8331.2411-0435. doi: 10.3778/j.issn.1002–8331.2411-0435.
|
| [10] |
LIU Chen, YANG Liying, LI Peike, et al. Dynamic derivation and elimination: Audio visual segmentation with enhanced audio semantics[C]. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, USA, 2025: 3131–3141. doi: 10.1109/CVPR52734.2025.00298.
|
| [11] |
VILAÇA L, YU Yi, and VIANA P. A survey of recent advances and challenges in deep audio-visual correlation learning[J]. ACM Computing Surveys, 2025, 57(12): 299. doi: 10.1145/3696445.
|
| [12] |
YE Qilang, YU Zitong, SHAO Rui, et al. CAT+: Investigating and enhancing audio-visual understanding in large language models[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025, 47(10): 8674–8690. doi: 10.1109/TPAMI.2025.3582389.
|
| [13] |
BANDARA T R, GOSTAR A K, ABEYSINGHE P, et al. Occlusion handling for intelligent vehicles using audio-visual multi-object tracking[J]. IEEE Transactions on Intelligent Transportation Systems, 2026, 27(9): 11578–11593. doi: 10.1109/TITS.2026.3697405.
|
| [14] |
TIAN Yapeng, SHI Jing, LI Bochen, et al. Audio-visual event localization in unconstrained videos[C]. 2018 15th European Conference on Computer Vision (ECCV), Munich, Germany, 2018: 252–268. doi: 10.1007/978-3-030-01216-8_16.
|
| [15] |
XU Haoming, ZENG Runhao, WU Qingyao, et al. Cross-modal relation-aware networks for audio-visual event localization[C]. 2020 28th ACM International Conference on Multimedia (ACM MM), Seattle, USA, 2020: 3893–3901. doi: 10.1145/3394171.3413581.
|
| [16] |
LIN Yanbo, LI Y J, and WANG Y C F. Dual-modality seq2seq network for audio-visual event localization[C]. 2019 44th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK, 2019: 2002–2006. doi: 10.1109/ICASSP.2019.8683226.
|
| [17] |
LIN Yanbo and WANG Y C F. Audiovisual transformer with instance attention for audio-visual event localization[C]. 2020 15th Asian Conference on Computer Vision (ACCV), Kyoto, Japan, 2021: 274–290. doi: 10.1007/978-3-030-69544-6_17.
|
| [18] |
ZHOU Jinxing, ZHENG Liang, ZHONG Yiran, et al. Positive sample propagation along the audio-visual event line[C]. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, USA, 2021: 8432–8440. doi: 10.1109/CVPR46437.2021.00833.
|
| [19] |
XIA Yan and ZHAO Zhou. Cross-modal background suppression for audio-visual event localization[C]. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, USA, 2022: 19957–19966. doi: 10.1109/CVPR52688.2022.01936.
|
| [20] |
XUAN Hanyu, ZHANG Zhenyu, CHEN Shuo, et al. Cross-modal attention network for temporal inconsistent audio-visual event localization[C]. 2020 34th AAAI Conference on Artificial Intelligence (AAAI-20), New York, USA, 2020: 279–286. doi: 10.1609/aaai.v34i01.5361.
|
| [21] |
SUN Chao, CHEN Min, CHENG Jialiang, et al. SCLAV: Supervised cross-modal contrastive learning for audio-visual coding[C]. 2023 31st ACM International Conference on Multimedia (ACM MM), Ottawa, Canada, 2023: 261–270. doi: 10.1145/3581783.3613805.
|
| [22] |
SUN Chao, CHEN Min, ZHU Chuanbo, et al. Listen with seeing: Cross-modal contrastive learning for audio-visual event localization[J]. IEEE Transactions on Multimedia, 2025, 27: 2650–2665. doi: 10.1109/TMM.2025.3535359.
|
| [23] |
YU Jiashuo, CHENG Ying, ZHAO Ruiwei, et al. MM-pyramid: Multimodal pyramid attentional network for audio-visual event localization and video parsing[C]. 2022 30th ACM International Conference on Multimedia (ACM MM), Lisbon, Portugal, 2022: 6241–6249. doi: 10.1145/3503161.3547869.
|
| [24] |
HE Xiang, LIU Xiangxi, LI Yang, et al. CACE-Net: Co-guidance attention and contrastive enhancement for effective audio-visual event localization[C]. 2024 32nd ACM International Conference on Multimedia (ACM MM), Melbourne, Australia, 2024: 985–993. doi: 10.1145/3664647.3681503.
|
| [25] |
GUZHOV A, RAUE F, HEES J, et al. Audioclip: Extending clip to image, text and audio[C]. 2022 47th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore, Singapore, 2022: 976–980. doi: 10.1109/ICASSP43922.2022.9747631.
|
| [26] |
LIN Yanbo, SUNG Y L, LEI Jie, et al. Vision transformers are parameter-efficient audio-visual learners[C]. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, Canada, 2023: 2299–2309. doi: 10.1109/CVPR52729.2023.00228.
|
| [27] |
WANG Kai, TIAN Yapeng, and DIMITRIOS H. Towards efficient audio-visual learners via empowering pre-trained vision transformers with cross-modal adaptation[C]. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, USA, 2024: 1837–1846. doi: 10.1109/CVPRW63382.2024.00190.
|
| [28] |
CHENG Ying, LI Yang, HE Junjie, et al. Mixture of experts for audio-visual learning[C]. Proceedings of the 38th International Conference on Neural Information Processing Systems, Vancouver, Canada, 2024: 7. doi: 10.52202/079017-0007.
|
| [29] |
HUANG Longzhuo, LI Liang, FU Xueyang, et al. Efficient prompt-based multimodal interaction for audio-visual event localization[C]. Proceedings of the 2025 International Conference on Multimedia Retrieval, Chicago, USA, 2025: 506–515. doi: 10.1145/3731715.3733320.
|