Semantic-guided Unified Multi-scale Deep Unrolling Network for Pansharpening

CHEN Junjie; WANG Tingting; FANG Faming; ZHANG Guixu

doi:10.11999/JEIT251252

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2026 >

CHEN Junjie, WANG Tingting, FANG Faming, ZHANG Guixu. Semantic-guided Unified Multi-scale Deep Unrolling Network for Pansharpening[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT251252

Citation:

CHEN Junjie, WANG Tingting, FANG Faming, ZHANG Guixu. Semantic-guided Unified Multi-scale Deep Unrolling Network for Pansharpening[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT251252

Citation:

PDF( 2576 KB)

Semantic-guided Unified Multi-scale Deep Unrolling Network for Pansharpening

doi: 10.11999/JEIT251252 cstr: 32379.14.JEIT251252

School of Computer Science and Technology, East China Normal University, Shanghai 200062, China

Funds: The National Key Research and Development Program of China (2022ZD0161800), The National Natural Science Foundation of China(62202173, 62271203), The Open Research Fund of KLATASDS-MOE, ECNU

Received Date: 2025-11-26
Accepted Date: 2026-04-17
Rev Recd Date: 2026-04-17

Available Online: 2026-05-03

Abstract

Abstract

Objective With the rapid development of satellite imaging technology, demand has increased for high-resolution multispectral remote sensing images in a wide range of applications. However, satellite platforms differ in sensor parameters and imaging conditions, which leads to clear domain shifts among datasets acquired by different satellites. Most existing Deep Learning (DL)-based pansharpening methods are therefore trained separately on individual satellite datasets and have limited cross-satellite generalization. To address this limitation, this study proposes a Semantic-guided Unified Multi-scale Deep Unrolling Network (SUM-DUN). SUM-DUN is designed based on classical optimization theory and adopts a three-dimensional (3D) multi-scale deep unrolling architecture for unified feature extraction and fusion. Multimodal Large Language Models (MLLMs) are used to generate semantic text prompts from the input images. These prompts guide the model to adaptively adjust feature representations and improve fusion quality. The proposed method aims to support unified remote sensing image fusion through a tailored network architecture and a prompt-guided mechanism, thereby providing reliable data for high-level image interpretation tasks. Methods Following the Maximum A Posteriori (MAP) estimation principle, the optimization process for High-Resolution Multispectral (HRMS) image recovery is unfolded into the proposed SUM-DUN (Fig. 1). Each iterative stage of SUM-DUN contains two main modules: a Gradient Descent Module (GDM) and a Semantic-guided Proximal Mapping Network (SPMN). These modules approximate the operations in Eq. (5) and Eq. (6), respectively. GDM performs gradient descent updating based on the current feature estimate and the degradation model. SPMN is implemented using a Transformer-based architecture, as shown in Fig. 2(b), and incorporates semantic text prompts generated from each input image pair by MLLMs. These prompts guide the network to select suitable feature propagation strategies for the current image pair. This process helps suppress noise and reduce discrepancies among different satellite sensors. Through upsampling and downsampling operations, the network also transmits multispectral (MS) and panchromatic (PAN) features across iterative stages. Thus, multi-scale spatial and spectral information is progressively preserved and enhanced during the deep unrolling process. Results and Discussions To verify the effectiveness of the proposed method, it is compared with seven representative baselines, including two traditional methods, BDSD and PRACS, and five DL-based methods, AWFLN, FusionMamba, PanMamba, WFANet, and TMDiff. In the reduced-resolution evaluation, ground-truth HRMS images are available. Several widely used reference-based metrics are adopted, including Spectral Angle Mapper (SAM), Spatial Correlation Coefficient (SCC), Peak Signal-to-Noise Ratio (PSNR), Erreur Relative Globale Adimensionnelle de Synthèse (ERGAS), Averaged Universal Image Quality Index (QAVE), and the Universal Image Quality Index for 4-band and 8-band images (Q4/Q8). These metrics jointly assess spectral fidelity, spatial consistency, and overall image quality. In the full-resolution evaluation, ground-truth HRMS images are unavailable. Therefore, no-reference quality indices are used. Specifically, Hybrid Quality with No Reference (HQNR), its spectral distortion component and spatial distortion component are used to assess fusion quality in real-world scenarios. Quantitative results on the GF-1, QB, WV-2, and WV-4 test datasets show that the proposed method consistently achieves the best or second-best performance across all metrics under both reduced-resolution and full-resolution settings (Tables 2 and 3). These results indicate that the proposed method can preserve spectral fidelity and spatial consistency while maintaining robust performance across different satellites and challenging imaging conditions. Ablation studies further validate the effectiveness of the 3D architecture, the multi-scale network design, and the spatial-channel prompt guidance mechanism. Removing or modifying any of these components causes performance degradation to different degrees (Tables 4 and 5). Conclusions This study proposes a semantic-guided unified multi-scale deep unrolling method for pansharpening. The method uses semantic prompts generated by an MLLM to support efficient and unified fusion of images from different satellites. The proposed approach is built on a deep unrolling framework and uses a 3D convolutional architecture to process satellite datasets with different numbers of spectral bands. A multi-scale network design is further used to extract spatial and spectral features at different levels, thereby improving fusion performance. In addition, a Semantic Prompt Integration Module (SPIM) is designed to adaptively route spatial and channel features based on semantic information. SPIM enables more effective feature propagation and improves both spatial detail reconstruction and spectral consistency. Extensive experiments show that the proposed method achieves state-of-the-art performance in visual quality and quantitative evaluation.
- Remote sensing image fusion,
- Unified pansharpening,
- Multimodal large language model,
- Deep unrolling network

FullText(HTML)

References(32)

References

[1]	THOMAS C, RANCHIN T, WALD L, et al. Synthesis of multispectral images to high spatial resolution: A critical review of fusion methods based on remote sensing physics[J]. IEEE Transactions on Geoscience and Remote Sensing, 2008, 46(5): 1301–1312. doi: 10.1109/TGRS.2007.912448.
[2]	金晶, 王峰. 分布式多卫星协同遥感图像场景分类方法[J]. 电子与信息学报, 2025, 47(12): 4677–4688. doi: 10.11999/JEIT250866. JIN Jing and WANG Feng. A distributed multi-satellite collaborative framework for remote sensing scene classification[J]. Journal of Electronics & Information Technology, 2025, 47(12): 4677–4688. doi: 10.11999/JEIT250866.
[3]	文泓力, 胡庆浩, 黄立威, 等. 基于参数高效ViT与多模态导引的遥感图像小样本分类方法[J]. 电子与信息学报, 2025, 47(12): 4689–4703. doi: 10.11999/JEIT250996. WEN Hongli, HU Qinghao, HUANG Liwei, et al. Few-shot remote sensing image classification based on parameter-efficient vision transformer and multimodal guidance[J]. Journal of Electronics & Information Technology, 2025, 47(12): 4689–4703. doi: 10.11999/JEIT250996.
[4]	韩汶杞, 蒋雯, 耿杰, 等. 原型对齐与拓扑一致性约束下的多模态半监督遥感图像语义分割[J]. 电子与信息学报, 2025, 47(12): 4714–4727. doi: 10.11999/JEIT251115. HAN Wenqi, JIANG Wen, GENG Jie, et al. PATC: Prototype alignment and topology-consistent pseudo-supervision for multimodal semi-supervised semantic segmentation of remote sensing images[J]. Journal of Electronics & Information Technology, 2025, 47(12): 4714–4727. doi: 10.11999/JEIT251115.
[5]	ZENG Delu, HU Yuwen, HUANG Yue, et al. Pan-sharpening with structural consistency and ℓ_1/2 gradient prior[J]. Remote Sensing Letters, 2016, 7(12): 1170–1179. doi: 10.1080/2150704X.2016.1222098.
[6]	WU Zhongcheng, HUANG Tingzhu, DENG Liangjian, et al. VO+Net: An adaptive approach using variational optimization and deep learning for panchromatic sharpening[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 5401016. doi: 10.1109/TGRS.2021.3066425.
[7]	LU Hangyuan, YANG Yong, HUANG Shuying, et al. AWFLN: An adaptive weighted feature learning network for pansharpening[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61: 5400815. doi: 10.1109/TGRS.2023.3241643.
[8]	XIE Xinyu, CUI Yawen, TAN Tao, et al. FusionMamba: Dynamic feature enhancement for multimodal image fusion with mamba[J]. Visual Intelligence, 2024, 2(1): 37. doi: 10.1007/s44267-024-00072-9.
[9]	HE Xuanhua, CAO Ke, ZHANG Jie, et al. Pan-mamba: Effective pan-sharpening with state space model[J]. Information Fusion, 2025, 115: 102779. doi: 10.1016/j.inffus.2024.102779.
[10]	HUANG Jie, HUANG Rui, XU Jinghao, et al. Wavelet-assisted multi-frequency attention network for pansharpening[C]. The 39th AAAI Conference on Artificial Intelligence, Philadelphia, USA, 2025: 3662–3670. doi: 10.1609/aaai.v39i4.32381.
[11]	JIA Menglin, TANG Luming, CHEN B C, et al. JIA Menglin, TANG Luming, CHEN B C, et al. Visual prompt tuning[C]. The 17th European Conference on Computer Vision, Tel-Aviv, Israel, 2022: 709–727. doi: 10.1007/978-3-031-19827-4_41.
[12]	NIE Xing, NI Bolin, CHANG Jianlong, et al. Pro-tuning: Unified prompt tuning for vision tasks[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2024, 34(6): 4653–4667. doi: 10.1109/TCSVT.2023.3327605.
[13]	CUI Yuning, ZAMIR S W, KHAN S, et al. AdaIR: Adaptive all-in-one image restoration via frequency mining and modulation[C]. The Thirteenth International Conference on Learning Representations, Singapore, 2025: 57335–57356.
[14]	10.1109/CVPR52734.2025.00705 10.1109/CVPR52734.2025.00705 ZENG Haijin, WANG Xiangming, CHEN Yongyong, et al. Vision-language gradient descent-driven all-in-one deep unfolding networks[C]. The 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, United States, 2025: 7524–7533.
[15]	YANG Zhiwen, CHEN Haowei, QIAN Ziniu, et al. All-in-one medical image restoration via task-adaptive routing[C]. 27th International Conference on Medical Image Computing and Computer Assisted Intervention, Marrakesh, Morocco, 2024: 67–77. doi: 10.1007/978-3-031-72104-5_7.
[16]	XING Yinghui, QU Litao, ZHANG Shizhou, et al. Empower generalizability for pansharpening through text-modulated diffusion model[J]. IEEE Transactions on Geoscience and Remote Sensing, 2024, 62: 5633812. doi: 10.1109/TGRS.2024.3434431.
[17]	LI Xueheng, HE Xuanhua, CAO Ke, et al. Exploring text-guided information fusion through chain-of-reasoning for pansharpening[J]. IEEE Transactions on Geoscience and Remote Sensing, 2025, 63: 5407314. doi: 10.1109/TGRS.2025.3604447.
[18]	FANG Shijie and GAN Hongping. SSUN-net: Spatial-spectral prior-aware unfolding network for pan-sharpening[C]. The 39th AAAI Conference on Artificial Intelligence, Philadelphia, United States, 2025: 2897–2905. doi: 10.1609/aaai.v39i3.32296.
[19]	YAN Qiuhai, JIANG Aiwen, CHEN Kang, et al. Textual prompt guided image restoration[J]. Engineering Applications of Artificial Intelligence, 2025, 155: 110981. doi: 10.1016/j.engappai.2025.110981.
[20]	CONDE M V, GEIGLE G, and TIMOFTE R. InstructIR: High-quality image restoration following human instructions[C]. The 18th European Conference on Computer Vision, Milan, Italy, 2024: 1–21. doi: 10.1007/978-3-031-72764-1_1.
[21]	ZENG Aohan, XU Bin, WANG Bowen, et al. ChatGLM: A family of large language models from GLM-130B to GLM-4 all tools[EB/OL]. https://arxiv.org/abs/2406.12793, 2024.
[22]	XIAO Shitao, LIU Zheng, ZHANG Peitian, et al. C-pack: Packed resources for general Chinese embeddings[C]. The 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, Washington, USA, 2024: 641–649. doi: 10.1145/3626772.3657878.
[23]	MENG Xiangchao, XIONG Yiming, SHAO Feng, et al. A large-scale benchmark data set for evaluating pansharpening performance: Overview and implementation[J]. IEEE Geoscience and Remote Sensing Magazine, 2021, 9(1): 18–52. doi: 10.1109/MGRS.2020.2976696.
[24]	WALD L, RANCHIN T, and MANGOLINI M. Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images[J]. Photogrammetric Engineering and Remote Sensing, 1997, 63(6): 691–699.
[25]	GARZELLI A, NENCINI F, and CAPOBIANCO L. Optimal MMSE pan sharpening of very high resolution multispectral images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2008, 46(1): 228–236. doi: 10.1109/TGRS.2007.907604.
[26]	CHOI J, YU Kiyun, and KIM Y. A new adaptive component-substitution-based satellite image fusion by using partial replacement[J]. IEEE Transactions on Geoscience and Remote Sensing, 2011, 49(1): 295–309. doi: 10.1109/TGRS.2010.2051674.
[27]	VIVONE G, ALPARONE L, CHANUSSOT J, et al. A critical comparison among pansharpening algorithms[J]. IEEE Transactions on Geoscience and Remote Sensing, 2015, 53(5): 2565–2586. doi: 10.1109/TGRS.2014.2361734.
[28]	ZHOU Jie, CIVCO D L, and SILANDER J A. A wavelet transform method to merge Landsat TM and SPOT panchromatic data[J]. International Journal of Remote Sensing, 1998, 19(4): 743–757. doi: 10.1080/014311698215973.
[29]	WALD L. Data Fusion. Definitions and Architectures - Fusion of Images of Different Spatial Resolutions[M]. Paris, France: Presses de l’École, Ecole des Mines de Paris, 2002: 165–189.
[30]	WANG Zhou and BOVIK A C. A universal image quality index[J]. IEEE Signal Processing Letters, 2002, 9(3): 81–84. doi: 10.1109/97.995823.
[31]	GARZELLI A and NENCINI F. Hypercomplex quality assessment of multi/hyperspectral images[J]. IEEE Geoscience and Remote Sensing Letters, 2009, 6(4): 662–665. doi: 10.1109/LGRS.2009.2022650.
[32]	ARIENZO A, VIVONE G, GARZELLI A, et al. Full-resolution quality assessment of pansharpening: Theoretical and hands-on approaches[J]. IEEE Geoscience and Remote Sensing Magazine, 2022, 10(3): 168–201. doi: 10.1109/MGRS.2022.3170092.