FD-Mamba：频域解耦状态空间模型驱动的可见光-红外行人重识别

李恪; 林淮荣; 童骏阳; 张杨; 王栋梁; 周开; 吴宋涛

doi:10.11999/JEIT260661

FD-Mamba：频域解耦状态空间模型驱动的可见光-红外行人重识别

doi: 10.11999/JEIT260661 cstr: 32379.14.JEIT260661

李恪^1,,
林淮荣¹,
童骏阳¹,
张杨²,
王栋梁^1, ,,
周开¹,
吴宋涛¹

1.
汕头大学工学院汕头 515063
2.
中国人民解放军海军研究院北京 100161

详细信息

作者简介:
李恪：男，副教授，研究方向为计算机视觉、群体智能邮箱：　ericlee@stu.edu.cn

林淮荣：男，硕士生，研究方向为计算机视觉、跨模态行人重识别

童骏阳：男，本科生，研究方向为模式识别

张杨：女，高级工程师，研究方向为计算机视觉

王栋梁：男，讲师，研究方向为群体智能邮箱：dlwang@stu.edu.cn

周开：男，硕士生，研究方向为群体智能

吴宋涛：男，硕士生，研究方向为计算机视觉

通讯作者:
王栋梁　dlwang@stu.edu.cn

中图分类号: TP391.4
计量
- 文章访问数: 13
- HTML全文浏览量: 2
- PDF下载量: 0
- 被引次数: 0
出版历程
- 收稿日期: 2026-05-20
- 修回日期: 2026-09-18
- 录用日期: 2026-09-28
- 网络出版日期: 2026-10-13

FD-Mamba:A Frequency-Decoupled State Space Model for Visible-Infrared Person Re-Identification

1.
College of Engineering, Shantou University, Shantou 515063, China
2.
Naval Research Institute of PLA, Beijing 100161, China

摘要

摘要: 针对可见光—红外行人重识别中跨模态光谱差异大、空间特征错位及现有方法过度依赖空间域建模的问题，该文提出频域解耦状态空间模型FD-Mamba。现有视觉Mamba方法主要面向通用单模态任务，缺少对跨模态行人结构与模态差异的针对性建模；现有频域方法多侧重频率响应建模，难以显式区分模态特异性噪声与共享结构信息。为此，FD-Mamba以双流ResNet-50为骨干，从空间域与频域协同增强跨模态表征。针对遮挡、姿态变化及局部特征错位导致的全局关系建模不足，设计混合空间Mamba注意力(HSMA)模块，通过多尺度卷积结合二维选择性扫描，以线性复杂度建立远距离身体区域依赖。针对高频模态噪声与低频共享结构相互纠缠的问题，构建跨尺度频域Mamba增强(CSFME)模块，利用多尺度离散余弦变换进行频率分解，并通过可学习频带门控与频率状态空间交互抑制高频模态干扰、增强低频共享结构。在SYSU-MM01、RegDB和LLCM数据集上的实验表明，FD-Mamba取得具有竞争力的性能，其中SYSU-MM01全搜索模式下Rank-1和mAP达到79.71%和77.52%。频谱统计、分频带识别、消融、运行效率及可视化分析进一步验证了频域解耦假设及HSMA与CSFME的有效性。
- 模式识别 /
- 可见光—红外行人重识别 /
- 跨模态检索 /
- 状态空间模型 /
- 频域解耦
Abstract: Objective Visible-infrared person re-identification (VI-ReID) aims to retrieve the same pedestrian identity across heterogeneous visible and infrared cameras, and is a key technique for all-day intelligent surveillance and cross-modal retrieval. The task remains challenging because visible images are formed by reflected light and carry rich color and texture, whereas infrared images are formed by thermal radiation and mainly describe body contours and thermal responses, which causes a large spectral gap and inconsistent local appearance. Viewpoint changes, pose variations, occlusion, and detection misalignment further weaken the spatial correspondence between body parts. Convolutional methods extract local cues effectively but describe global body topology poorly, while Transformer-based methods model long-range dependencies at a quadratic cost that limits practical deployment. More importantly, most existing methods align features only in the spatial domain and ignore the different roles of frequency components: low-frequency components preserve the body shape and structural layout shared by both modalities, whereas high-frequency components mainly carry visible texture, illumination edges, infrared thermal noise, and other modality-specific disturbances. To address these limitations, this paper proposes a frequency-decoupled state space model, termed FD-Mamba, that jointly enhances spatial structural modeling and frequency-domain cross-modal alignment. The proposed design is further motivated by the observation that cross-modal discrepancy does not affect all frequency components equally. Separating shared structural information from modality-specific responses therefore provides a more targeted way to improve modality invariance than treating all spectral components uniformly. Methods As illustrated in Fig. 1, FD-Mamba is built upon a dual-stream ResNet-50 backbone. Shallow stages use modality-specific branches to preserve modality-dependent low-level cues, while deeper stages share weights to project both modalities into a common semantic space. Two modules are introduced. First, a Hybrid Spatial Mamba Attention (HSMA) module is designed for deep semantic features, as shown in Fig. 3. HSMA combines multi-scale convolution with two-dimensional selective scanning: the convolution branch extracts local body-part patterns under different receptive fields, while the scanning branch propagates contextual information along multiple spatial directions, following the selective scan of Fig. 2, so that long-range interactions among the head, torso, legs, and carried objects are captured and more complete identity-related representations are obtained without relying only on isolated local textures. Second, a Cross-Scale Frequency Mamba Enhancement (CSFME) module is constructed for shallow and intermediate features, as shown in Fig. 4. CSFME divides feature channels into groups and projects them into the frequency domain through multi-scale two-dimensional Discrete Cosine Transform. The DCT coefficients of each group are partitioned into zigzag-ordered frequency bands and reweighted by a learnable band-gating mechanism, so that high-frequency modality noise is suppressed softly rather than truncated. The reweighted bands are then coupled by a frequency-aware state-space process, transformed back to the spatial domain by inverse DCT, and recalibrated by pixel and channel attention. Modality-specific noise is thus suppressed and low-frequency shared structural cues enhanced, providing more robust features for subsequent spatial alignment. The whole network is trained end-to-end with the identity, metric, and prototype-related constraints inherited from the baseline protocol. Both modules are inserted into the original backbone in a residual manner, so the enhanced representations retain the original identity information while introducing complementary spatial and frequency cues. This design also limits the additional computational cost and preserves the overall training protocol of the baseline. Results and Discussions Experiments are conducted on three representative VI-ReID benchmarks, SYSU-MM01, RegDB, and LLCM, which cover large-scale cross-camera retrieval, bidirectional visible-infrared matching, and low-light cross-modal scenarios, respectively. As reported in Table 1, FD-Mamba achieves 79.71% Rank-1 and 77.52% mAP on SYSU-MM01 under the all-search mode, improving the strong PDM baseline by 2.76 and 3.27 percentage points and indicating that the spatial-frequency collaborative design benefits both top-ranked accuracy and overall ranking quality. Competitive performance is also obtained under indoor-search. On RegDB, FD-Mamba achieves 95.2% Rank-1 and 89.3% mAP in the VIS to IR direction, and 94.7% Rank-1 and 87.5% mAP in the IR to VIS direction. The close results in both directions suggest that the model learns a relatively symmetric cross-modal representation rather than relying on a fixed query modality. The retrieval examples in Fig.5(b) further show that FD-Mamba reduces incorrect top-ranked matches compared with the baseline model. As shown in Table 2, FD-Mamba remains stable on the more challenging LLCM dataset, obtaining 65.3% Rank-1 and 67.8% mAP for Visible-to-Infrared retrieval and 57.5% Rank-1 and 63.9% mAP for Infrared-to-Visible retrieval, indicating robustness under low illumination, background clutter, and local texture degradation. Ablation studies in Tables 4 and 5 show that HSMA improves Rank-1 and mAP by 1.15 and 2.50 percentage points and CSFME by 0.90 and 1.25 percentage points, while their combination performs best, demonstrating the complementarity between global spatial topology modeling and frequency-domain noise suppression. Spectral statistics and band-limited recognition experiments quantitatively support the frequency-decoupling assumption. Additional analyses show that the gains are not produced by a single component. The frequency-domain experiments indicate that low-frequency information is more stable across modalities, whereas high-frequency responses contain stronger modality-specific variations, which is consistent with the improvements brought by learnable frequency-band reweighting. FD-Mamba contains 42.6M parameters and 8.51G FLOPs, only 1.83% and 2.90% more than the baseline, with comparable throughput and latency on an RTX 4090 GPU. The attention maps in Fig. 5(a) show more continuous pedestrian foreground responses than the baseline, and the feature distance distributions and t-SNE visualization in Fig. 6 further reveal smaller intra-class cross-modal distances and clearer inter-class margins, indicating improved identity discrimination and modality consistency. Conclusions This paper presents FD-Mamba, a frequency-decoupled state space model for visible-infrared person re-identification. The HSMA module improves global body-structure modeling through hybrid spatial scanning, and the CSFME module introduces explicit frequency decomposition to suppress modality-specific disturbances while preserving shared structural information. Quantitative comparisons, ablation studies, and visual analyses consistently show that the proposed spatial-frequency collaborative framework improves both identity discrimination and cross-modal consistency, making it suitable for all-day cross-modal pedestrian retrieval and suggesting that frequency-aware state-space modeling is a promising direction for heterogeneous visual representation learning.The current framework is still evaluated mainly under supervised benchmark protocols. Future work will investigate unsupervised cross-modal settings and more lightweight state-space designs to further improve generalization and deployment efficiency in practical surveillance scenarios.
- Pattern recognition /
- Visible-infrared person re-identification /
- Cross-modal retrieval /
- State space model /
- Frequency decoupling

HTML全文

图 1 FD-Mamba框架的整体架构

下载: 全尺寸图片幻灯片

图 2 二维选择性扫描(SS2D)机制示意图

下载: 全尺寸图片幻灯片

图 3 HSMA框架图

下载: 全尺寸图片幻灯片

图 4 跨尺度频域Mamba增强框架图

下载: 全尺寸图片幻灯片

图 5 可视化结果

下载: 全尺寸图片幻灯片

图 6 跨模态特征距离分布与t-SNE可视化对比。

下载: 全尺寸图片幻灯片

表 5 HSMA/CSFME组件与插入位置消融(SYSU-MM01，all-search)

分组	配置	Rank-1(%)	mAP(%)
HSMA	Baseline+HSMA(完整)	78.10	76.75
	w/o 多尺度卷积分支	77.3	75.62
	w/o SS2D	77.08	75.21
	w/o 门控FFN	77.62	76.08
	级联双7×7替换为单个13×13	77.94	76.41
CSFME	Baseline+CSFME (G=4, K=4，默认)	77.85	75.5
	G=1	77.21	74.83
	G=8	77.68	75.31
	K=2	77.49	75.08
	w/o 频带门控	77.28	74.91
	w/o SS2D频率交互	77.36	75.02
插入位置	CSFME@stage1+2， HSMA@stage4(默认)	79.71	77.52
	CSFME仅@stage1	79.02	76.84
	CSFME@stage2+3	78.63	76.4
	HSMA@stage3	79.12	76.9
	HSMA@stage3+4	79.53	77.38
	stage1处CSFME改为双流共享参数	79.33	77.14

下载: 导出CSV

表 1 在SYSU-MM01与RegDB数据集上与最先进方法的比较

数据集		SYSU-MM01^[19]				RegDB^[20]
数据集		All-search		Indoor-search		VIStoIR		IRtoVIS
方法	Venue	Rank-1(%)	mAP(%)	Rank-1(%)	mAP(%)	Rank-1(%)	mAP(%)	Rank-1(%)	mAP(%)
DDAG^[2]	ECCV'20	54.8	53	61	68	69.3	63.5	68.1	61.8
AGW^[1]	TPAMI'21	47.5	47.7	54.2	61.5	70	66.4	70.5	65.9
TransReID^[3]	ICCV'21	68.5	66.3	75.2	78.1	81.2	77.5	80.8	76.9
DEEN^[6]	CVPR'23	72.2	70.1	79.5	82.3	84.5	80.5	83.2	79
MIP^[22]	ICMR'24	70.8	66.4	78.8	79.9	91.3	85.9	92.4	86
IDKL^[23]	CVPR'24	75.8	73.1	84.2	86.8	87.5	83.2	86.8	82.5
ACD^[24]	TIFS'24	74.4	71.2	78.9	82.8	84.7	83.3	87.2	84.7
CSDN^[25]	TMM'25	75.2	71.8	82	85	89	76.6	84.2	75.1
MPT^[26]	ICMR'25	78.6	74.2	83.1	87.5	95.1	89.6	94.3	89
PDM(基线)^[7]	ICASSP'25	76.9	74.3	85.1	87.6	93.4	87.2	92.8	86.1
FD-Mamba	本文方法	79.7	77.5	87.4	89.7	95.2	89.3	94.7	87.5

下载: 导出CSV

表 2 在LLCM数据集上与最先进方法的比较(FD-Mamba为3个随机种子的均值±标准差)

Methods	VIS to IR		IR to VIS
Methods	mAP(%)	Rank-1(%)	mAP(%)	Rank-1(%)
DDAG^[2]	48.4	40.3	52.3	48
DEEN^[6]	65.8	62.5	62.9	54.9
ARGN^[27]	66.6	63.9	63.3	56.9
AMML^[28]	53.4	68.3	60.8	54.3
PDM(Baseline)^[7]	67.3	64.9	63.6	57.1
FD-Mamba(本文方法)	67.8±0.3	65.3±0.4	63.9±0.3	57.5±0.5

下载: 导出CSV

表 3 分频带识别实验与跨模态频带统计(SYSU-MM01，all-search, single-shot)

输入频带	Rank-1(%)	mAP(%)	跨模态余弦距离
完整输入	76.95	74.25	0.31
仅低频	68.52	65.87	0.18
仅中频	55.31	52.68	0.34
仅高频	41.23	38.62	0.52

下载: 导出CSV

表 4 在SYSU-MM01数据集上的消融研究(all-search模式)

Baseline	HSMA	CSFME	SYSU-MM01^[19]
Baseline	HSMA	CSFME	Rank-1 (%)	mAP (%)
√			76.95	74.25
√	√		78.10	76.75
√		√	77.85	75.50
√	√	√	79.71	77.52

下载: 导出CSV

表 6 与Transformer及视觉Mamba方法的精度与运行效率比较(RTX 4090，输入3×384×192，SYSU-MM01 all-search)

方法	参数量(M)	FLOPs(G)	吞吐量(img/s)	延迟(ms)	显存(GB)	Rank-1(%)	mAP(%)
TransReID^[3]	86.6	18.5	512	19.8	8.7	68.5	66.3
VMamba-T双流^[9]	31.2	5.7	986	11.2	5.3	74.82	72.1
Vision Mamba-S双流^[10]	26	5.1	1015	10.8	5.1	73.94	71.25
PDM(基线)^[7]	41.8	8.27	1265	8.9	4.6	76.95	74.25
FD-Mamba(MSM替换为VMamba块)	42.4	8.46	1190	9.5	4.9	78.91	76.75
FD-Mamba(本文)	42.6	8.51	1173	9.6	4.9	79.71	77.52

下载: 导出CSV

参考文献(30)

[1]	YE Mang, SHEN Jianbing, LIN Gaojie, et al. Deep learning for person re-identification: A survey and outlook[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(6): 2872–2893. doi: 10.1109/TPAMI.2021.3054775.
[2]	YE Mang, SHEN Jianbing, CRANDALL D J, et al. Dynamic dual-attentive aggregation learning for visible-infrared person re-identification[C]. Proceedings of the 16th European Conference on Computer Vision-ECCV 2020, Glasgow, UK, 2020: 229–247. doi: 10.1007/978-3-030-58520-4_14.
[3]	HE Shuting, LUO Hao, WANG Pichao, et al. TransReID: Transformer-based object re-identification[C]. Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, Canada, 2021: 14993–15002. doi: 10.1109/ICCV48922.2021.01474.
[4]	YU Hao, CHENG Xu, PENG Wei, et al. Modality unifying network for visible-infrared person re-identification[C]. Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France, 2023: 11151–11161. doi: 10.1109/ICCV51070.2023.01027.
[5]	张红颖, 樊世钰, 罗谦, 等. 结合视觉文本匹配和图嵌入的可见光-红外行人重识别[J]. 电子与信息学报, 2024, 46(9): 3662–3671. doi: 10.11999/JEIT240318. ZHANG Hongying, FAN Shiyu, LUO Qian, et al. Visible-infrared person re-identification combining visual-textual matching and graph embedding[J]. Journal of Electronics & Information Technology, 2024, 46(9): 3662–3671. doi: 10.11999/JEIT240318.
[6]	ZHANG Yukang and WANG Hanzi. Diverse embedding expansion network and low-light cross-modality benchmark for visible-infrared person re-identification[C]. Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, Canada, 2023: 2153–2162. doi: 10.1109/CVPR52729.2023.00214.
[7]	LI Jiarui, QIU Zhen, YANG Yilin, et al. Prototype-driven multi-feature generation for visible-infrared person re-identification[C]. Proceedings of the ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Hyderabad, India, 2025: 1–5. doi: 10.1109/ICASSP49660.2025.10889917.
[8]	GU A and DAO T. Mamba: Linear-time sequence modeling with selective state spaces[C]. Proceedings of the 1th Conference on Language Modeling, Philadelphia, US, 2024.
[9]	LIU Yue, TIAN Yunjie, ZHAO Yuzhong, et al. VMamba: Visual state space model[C]. Proceedings of the 38th International Conference on Neural Information Processing Systems, Vancouver, Canada, 2024: 3273.
[10]	ZHU Lianghui, LIAO Bencheng, ZHANG Qian, et al. Vision mamba: Efficient visual representation learning with bidirectional state space model[C]. Proceedings of the 41st International Conference on Machine Learning, Vienna, Austria, 2024: 2584.
[11]	WANG Zeyu, LI Chen, XU Huiying, et al. Mamba YOLO: A simple baseline for object detection with state space model[C]. Proceedings of the 39th Annual AAAI Conference on Artificial Intelligence, Philadelphia, USA, 2025: 8205–8213. doi: 10.1609/aaai.v39i8.32885.
[12]	ZHOU Mingliang, LENG Hongyue, FANG Bin, et al. Low-light image enhancement via a frequency-based model with structure and texture decomposition[J]. ACM Transactions on Multimedia Computing, Communications, and Applications, 2023, 19(6): 187. doi: 10.1145/3590965.
[13]	QIN Zequn, ZHANG Pengyi, WU Fei, et al. FcaNet: Frequency channel attention networks[C]. Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, Canada, 2021: 763–772. doi: 10.1109/ICCV48922.2021.00082.
[14]	NAM J, SYAZWANY N S, KIM S J, et al. Modality-agnostic domain generalizable medical image segmentation by multi-frequency in multi-scale attention[C]. Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, USA, 2024: 11480–11491. doi: 10.1109/CVPR52733.2024.01091.
[15]	HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, USA, 2016: 770–778. doi: 10.1109/CVPR.2016.90.
[16]	AHMED N, NATARAJAN T, and RAO K R. Discrete cosine transform[J]. IEEE Transactions on Computers, 1974, C-23(1): 90–93. doi: 10.1109/T-C.1974.223784.
[17]	王粉花, 赵波, 黄超, 等. 基于多尺度和注意力融合学习的行人重识别[J]. 电子与信息学报, 2020, 42(12): 3045–3052. doi: 10.11999/JEIT190998. WANG Fenhua, ZHAO Bo, HUANG Chao, et al. Person re-identification based on multi-scale network attention fusion[J]. Journal of Electronics & Information Technology, 2020, 42(12): 3045–3052. doi: 10.11999/JEIT190998.
[18]	庄建军, 庄宇辰. 一种结构化双注意力混合通道增强的跨模态行人重识别方法[J]. 电子与信息学报, 2024, 46(2): 518–526. doi: 10.11999/JEIT230614. ZHUANG Jianjun and ZHUANG Yuchen. A cross-modal person re-identification method based on hybrid channel augmentation with structured dual attention[J]. Journal of Electronics & Information Technology, 2024, 46(2): 518–526. doi: 10.11999/JEIT230614.
[19]	WU Ancong, ZHENG Weishi, YU Hongxing, et al. RGB-infrared cross-modality person re-identification[C]. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 2017: 5390–5399. doi: 10.1109/ICCV.2017.575.
[20]	ZHENG Weishi, GONG Shaogang, and XIANG Tao. Person re-identification by probabilistic relative distance comparison[C]. Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Colorado Springs, USA, 2011: 649–656. doi: 10.1109/CVPR.2011.5995598.
[21]	PASZKE A, GROSS S, MASSA F, et al. PyTorch: An imperative style, high-performance deep learning library[C]. Proceedings of the 33rd International Conference on Neural Information Processing Systems, Vancouver, Canada, 2019: 721.
[22]	WU Ruiqi, JIAO Bingliang, WANG Wenxuan, et al. Enhancing visible-infrared person re-identification with modality- and instance-aware visual prompt learning[C]. Proceedings of the 2024 International Conference on Multimedia Retrieval, Phuket, Thailand, 2024: 579–588. doi: 10.1145/3652583.3658109.
[23]	REN Kaijie and ZHANG Lei. Implicit discriminative knowledge learning for visible-infrared person re-identification[C]. Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, USA, 2024: 393–402. doi: 10.1109/CVPR52733.2024.00045.
[24]	PAN Honghu, PEI Wenjie, LI Xin, et al. Unified conditional image generation for visible-infrared person re-identification[J]. IEEE Transactions on Information Forensics and Security, 2024, 19: 9026–9038. doi: 10.1109/TIFS.2024.3426335.
[25]	YU Xiaoyan, DONG Neng, ZHU Liehuang, et al. CLIP-driven semantic discovery network for visible-infrared person re-identification[J]. IEEE Transactions on Multimedia, 2025, 27: 4137–4150. doi: 10.1109/TMM.2025.3535353.
[26]	ZHANG Hongde and MA Bingpeng. Visible-infrared person re-identification via mutual reinforcement of prompts and image encoders[C]. Proceedings of the 2025 International Conference on Multimedia Retrieval, Chicago, USA, 2025: 2118–2122. doi: 10.1145/3731715.3733496.
[27]	QI Mengzan, CHAN Sixian, HANG Chen, et al. Auxiliary representation guided network for visible-infrared person re-identification[J]. IEEE Transactions on Multimedia, 2025, 27: 340–355. doi: 10.1109/TMM.2024.3521773.
[28]	ZHANG Yukang, YAN Yan, LU Yang, et al. Adaptive middle modality alignment learning for visible-infrared person re-identification[J]. International Journal of Computer Vision, 2025, 133(4): 2176–2196. doi: 10.1007/s11263-024-02276-4.
[29]	VAN DER MAATEN L and HINTON G. Visualizing data using t-SNE[J]. Journal of Machine Learning Research, 2008, 9: 2579–2605.
[30]	崔学英, 王宇航, 刘斌, 等. Wave-MambaCT: 基于小波Mamba的低剂量CT伪影抑制方法[J]. 电子与信息学报, 2026, 48(3): 982–993. doi: 10.11999/JEIT250489. CUI Xueying, WANG Yuhang, LIU Bin, et al. Wave-MambaCT: Low-dose CT artifact suppression method based on wavelet Mamba[J]. Journal of Electronics & Information Technology, 2026, 48(3): 982–993. doi: 10.11999/JEIT250489.