SO(3)流形约束的双鱼眼全景图像配准方法

王卓鹏; 林珊玲; 林坚普; 吕珊红; 林志贤

doi:10.11999/JEIT260798

SO(3)流形约束的双鱼眼全景图像配准方法

doi: 10.11999/JEIT260798 cstr: 32379.14.JEIT260798

王卓鹏^1,,
林珊玲¹,
林坚普¹,
吕珊红¹,
林志贤^{1, 2, ,}

1.
福州大学先进制造学院泉州 362251
2.
中国福建光电信息科学与技术创新实验室福州 350116

基金项目: 国家重点研发计划课题(2021YFB3600603)

详细信息

作者简介:
王卓鹏：男，硕士生，研究方向为双鱼眼全景图像配准、全景视觉，邮箱：　19967297501@163.com

林珊玲：女，博士，副教授，硕士生导师，研究方向为显示驱动、图像处理

林坚普：男，博士，副教授，硕士生导师，研究方向为图像处理、深度学习、液晶透镜器件、裸眼3D显示技术

吕珊红：女，讲师，硕士生导师，研究方向为量子点、显示驱动技术

林志贤：男，教授，博士生导师，研究方向为图像处理、平板显示驱动技术，邮箱：　lzx2005000@163.com

通讯作者:
林志贤　lzx2005000@163.com

中图分类号: TN911.73; TP391.41
计量
- 文章访问数: 35
- HTML全文浏览量: 9
- PDF下载量: 11
- 被引次数: 0
出版历程
- 收稿日期: 2026-06-16
- 修回日期: 2026-08-11
- 录用日期: 2026-08-13
- 网络出版日期: 2026-08-15

An SO(3)-Manifold-Constrained Registration Method for Twin-Fisheye Panoramic Images

1.
School of Advanced Manufacturing, Fuzhou University, Quanzhou 362251, China
2.
Fujian Science & Technology Innovation Laboratory for Optoelectronic Information of China, Fuzhou 350116, China

Funds: The National Key Research and Development Program of China (Grant No. 2021YFB3600603)

摘要

摘要: 针对双鱼眼图像经等距柱状投影(Equirectangular Projection, ERP)展开后存在重叠带狭长、左右周期边界处不连续、以及平面单应模型在球面成像下引入冗余自由度而放大噪声等问题，该文提出一种基于SO(3)流形约束的双鱼眼全景图像配准方法。首先，设计ERP重叠带感兴趣区域(Region of Interest, ROI)提取与边界环绕机制，通过经度方向循环重排恢复跨边界的特征连续性；其次，构建三维旋转群SO(3)上的鲁棒估计模块，以球面角残差替代平面像素残差作为内点判据，消除冗余自由度；最后，在李代数so(3)上进行自适应非线性精化以抑制长尾残差。在PanoraMIS数据集上，以序列帧间相对旋转为定量评测，该方法配准成功率达97.10%，P95角残差0.549°，时序稳定性0.313°，较SuperPoint+LightGlue在P95残差和时序稳定性上分别降低22.8%和77.3%，较Efficient LoFTR峰值显存降低52.7%。在配准精度、时序稳定性与资源开销之间取得了良好平衡，具备面向端侧双鱼眼全景拼接前端的部署潜力。
- 双鱼眼图像 /
- 全景拼接 /
- 等距柱状投影 /
- 轻量特征匹配 /
- SO(3)-RANSAC
Abstract: Objective Twin-fisheye cameras provide near-360° coverage with low hardware complexity and are widely used in immersive imaging, surveillance, and mobile robotics. Their panoramic output depends on registration over a narrow overlapping band, so geometric accuracy and temporal consistency directly affect seam quality and video smoothness. After the two fisheye views are unfolded into the Equirectangular Projection (ERP), three coupled problems arise. First, the near-co-centric lens pair is ideally related by a pure rotation R ∈SO(3), whereas a conventional 8-Degree-of-Freedom (DoF) homography introduces five redundant parameters that may couple with matching noise. Second, ERP sampling is nonuniform with latitude, so identical pixel residuals do not represent identical spherical angular errors. Third, the cyclic ±π longitude boundary splits structures that are continuous on the sphere and weakens correspondences around the seam. Existing planar pipelines and generic learned matchers rarely combine these constraints under a unified rotation-referenced evaluation. This study therefore develops a lightweight registration framework that explicitly exploits spherical rotation geometry while addressing ERP boundary discontinuity, temporal fluctuation, and long-tail residuals. Methods The proposed framework contains three modules (Fig. 1). First, an overlapping-band Region-of-Interest (ROI) is cropped around the ERP seam and rearranged with modulo-W wrap-around (Fig. 2). A default longitude half-width of ±15° and an approximately 3° margin on each side preserve cross-boundary feature continuity while restricting the search region. XFeat detects, describes, and matches features under a fixed Top-K budget. Second, the two-dimensional matches are restored to global ERP coordinates, mapped to unit-sphere direction vectors, and processed by rotation-only SO(3)-RANSAC. Spherical angular residuals are used as the inlier criterion with a 0.8° threshold, a maximum of 2000 iterations, confidence 0.999, and at least 12 inliers; the iteration bound is updated adaptively. All inliers are then used for Kabsch/SVD closed-form rotation re-estimation, which reduces the randomness of a minimal sample while preserving the SO(3) constraint. Third, a local increment on the Lie algebra so(3) is optimized under a Huber loss by the Levenberg–Marquardt algorithm. The refinement is triggered only when the inlier-residual P95 exceeds 0.90° and the inlier ratio is below 0.58, thereby concentrating nonlinear optimization on difficult image pairs. Results and Discussions Experiments are conducted on PanoraMIS Sequences 3 and 4 under a unified relative inter-frame rotation protocol (Table 1). The proposed method achieves a 97.10% success rate, a 0.549° P95 angular residual, and a 0.313° temporal-stability error. Compared with SuperPoint+LightGlue, the P95 and temporal-stability errors are reduced by 22.8% and 77.3%, respectively. Compared with Efficient LoFTR, peak GPU memory and runtime are reduced by 52.7% and 60.9%, although Efficient LoFTR retains the lowest overall P95. Under a unified SO(3)-RANSAC back-end (Table 2), XFeat provides the largest average inlier count of 625.4 and the lowest temporal-stability error of 0.313° at 38.04 ms. The ablation and sensitivity results (Tables 3–4) show that the ±15° ROI reduces the P95 from 0.720° for the full ERP to 0.600°. Replacing H-RANSAC with SO(3)-RANSAC reduces temporal instability from 0.931° to 0.313°, a 66.3% reduction, while increasing runtime from 24.91 ms to 35.24 ms. Adaptive refinement operates on approximately one third of the image pairs and improves both P95 and temporal stability with lower overhead than always-on refinement; its five-seed mean and median trigger rate are both 36.23%. A Top-K budget of 2048 reaches the saturated accuracy level, because increasing the budget to 4096 yields no further P95 or stability improvement. Five fixed-seed repetitions produce standard deviations no greater than 0.011° for P95 and stability, indicating that the main conclusions are insensitive to RANSAC randomness. In a 5×5 threshold sweep, the maximum changes in P95 and inter-frame rotation jitter within the central neighborhood are 4.40% and 0.037%, respectively, and changing the robust-error truncation from 3° to 2° or 5° does not alter the relative ranking. On the more difficult Sequence 4, characterized by weak texture and unstable overlap, the proposed method obtains 919.7 average inliers and a P95 of 0.383°, the lowest among the evaluated learning-based matchers, although robust-estimation time increases. With an identical standardized stitching back-end, it produces a lower seam-band gradient than H-RANSAC in the small-rotation example (26.90 versus 28.71) and a lower truth-referenced temporal-stability error (0.313° versus 0.931°; Fig. 6). On outdoor Sequence 7-L2, 324 of 346 correspondences are retained, yielding a 93.6% inlier ratio and a 0.524° P95 residual (Fig. 5). Because sequence-specific calibration is unavailable and the fixed inter-lens baseline may cause depth-dependent parallax, this result serves only as a diagnostic consistency check, not as evidence of absolute pose accuracy. Conclusions By restoring feature continuity across the ERP boundary, replacing the redundant planar homography with an explicit SO(3) rotation model, and selectively refining difficult image pairs on so(3), the proposed method balances registration accuracy, temporal consistency, and resource cost. It provides a lightweight front-end for twin-fisheye panorama stitching. The current evaluation is limited to pairwise registration on a small number of sequences; future work will address translation compensation, multi-frame global optimization, end-to-end integration with seam finding, exposure compensation, and blending, as well as generalization across additional platforms, dynamic scenes, and illumination conditions.
- Twin-fisheye images /
- Panoramic stitching /
- Equirectangular projection /
- Lightweight feature matching /
- SO(3)-RANSAC

HTML全文

图 1 本文方法总体框架

下载: 全尺寸图片幻灯片

图 2 ROI裁剪与边界环绕拼接示意图

下载: 全尺寸图片幻灯片

图 3 不同方法的内点角残差CDF曲线

下载: 全尺寸图片幻灯片

图 4 Sequence 3和Sequence 4上的原始匹配与SO(3)-RANSAC内点筛选结果

下载: 全尺寸图片幻灯片

图 5 Sequence 7-L2上的匹配与SO(3)-RANSAC内点筛选结果

下载: 全尺寸图片幻灯片

图 6 标准化后端下H-RANSAC与SO(3)-RANSAC的拼接对比

下载: 全尺寸图片幻灯片

表 1 不同方法在PanoraMIS数据集上的对比结果

序列	方法	Median/(°)↓	P95/(°)↓	#Inliers↑	Success/%↑	Stability/(°)↓	Time/ms↓	Peak VRAM/MB↓
Seq 3	ORB	0.221	0.734	115.7	88.14	0.930	19.89	0.0
	SIFT	0.094	0.735	65.7	79.66	0.971	66.00	0.0
	SP+LG	0.323	0.722	243.3	100.00	1.372	28.81	216.7
	E-LoFTR	0.179	0.390	424.1	100.00	0.290	103.15	364.0
	Ours	0.228	0.572	575.5	100.00	0.138	30.03	172.1
Seq 4	ORB	0.003	0.306	761.8	70.00	1.466	20.80	0.0
	SIFT	0.021	0.194	246.1	80.00	1.462	68.09	0.0
	SP+LG	0.122	0.513	353.6	80.00	1.424	40.23	216.7
	E-LoFTR	0.120	0.670	373.4	70.00	1.348	48.83	364.0
	Ours	0.084	0.383	919.7	80.00	1.345	79.59	172.1
Overall	ORB	0.103	0.661	209.4	85.51	1.008	20.02	0.0
	SIFT	0.060	0.686	91.8	79.71	1.042	66.30	0.0
	SP+LG	0.281	0.711	259.3	97.10	1.379	30.47	216.7
	E-LoFTR	0.173	0.436	416.8	95.65	0.444	95.28	364.0
	Ours	0.196	0.549	625.4	97.10	0.313	37.22	172.1

下载: 导出CSV

表 2 统一SO(3)-RANSAC后端下的前端公平性对比

前端方法	Median/(°)↓	P95/(°)↓	#Inliers↑	Inlier Ratio↑	Success/%↑	Stability/(°)↓	Time/ms↓
ORB	0.201	0.538	265.6	0.663	97.10	1.183	47.35
SIFT	0.259	0.661	116.0	0.556	98.55	1.041	98.34
SP+LG	0.281	0.711	259.3	0.559	97.10	1.379	30.91
E-LoFTR	0.173	0.436	416.8	0.674	95.65	0.444	99.10
XFeat	0.196	0.549	625.4	0.620	97.10	0.313	38.04
注：表2耗时为不含显存插桩的独立计时结果，显存及含插桩耗时以表1为准。

下载: 导出CSV

表 3 关键模块消融实验结果

实验项	设置	P95/(°)↓	Success/%↑	Stability/(°)↓	Time/ms↓	Trigger/%
ROI	Full ERP	0.720	98.55	0.249	28.52	—
	ROI ±10°	0.631	95.65	0.528	29.77	—
	ROI ±15°	0.600	95.65	0.284	23.47	—
	ROI ±20°	0.660	95.65	0.446	20.35	—
	ROI ±30°	0.665	95.65	0.516	19.11	—
RANSAC	H-RANSAC	0.579	97.10	0.931	24.91	—
RANSAC	SO(3)-RANSAC	0.549	97.10	0.313	35.24	—
Refine	No refine	0.600	95.65	0.284	22.81	0.00
	Always refine	0.543	95.65	0.202	29.31	100.00
	Adaptive refine	0.587	95.65	0.272	24.95	34.78
注：“—”表示该指标不适用或未统计。RANSAC组采用2048×1024分辨率，ROI组、Refine组及表4采用1024×512分辨率；ROI组和表4均关闭非线性精化。Adaptive refine的Trigger为种子20260706下的结果，5个固定种子下的平均和中位触发率均为36.23%，范围为34.78%～39.13%。

下载: 导出CSV

表 4 关键点预算K的敏感性实验结果

K	P95/(°)↓	Success/%↑	Stability/(°)↓	Time/ms↓
512	0.688	95.65	0.593	28.83
1024	0.622	95.65	0.292	25.45
2048	0.600	95.65	0.284	22.79
4096	0.600	95.65	0.284	22.75

下载: 导出CSV

参考文献(23)

[1]	SZELISKI R. Image alignment and stitching: A tutorial[J]. Foundations and Trends^® in Computer Graphics and Vision, 2006, 2(1): 1–104. doi: 10.1561/0600000009.
[2]	BENSEDDIK H E, MORBIDI F, and CARON G. PanoraMIS: An ultra-wide field of view image dataset for vision-based robot-motion estimation[J]. The International Journal of Robotics Research, 2020, 39(9): 1037–1051. doi: 10.1177/0278364920915248.
[3]	LOWE D G. Distinctive image features from scale-invariant keypoints[J]. International Journal of Computer Vision, 2004, 60(2): 91–110. doi: 10.1023/B:VISI.0000029664.99615.94.
[4]	RUBLEE E, RABAUD V, KONOLIGE K, et al. ORB: An efficient alternative to SIFT or SURF[C]. 2011 International Conference on Computer Vision, Barcelona, Spain, 2011: 2564–2571. doi: 10.1109/ICCV.2011.6126544.
[5]	DETONE D, MALISIEWICZ T, and RABINOVICH A. SuperPoint: Self-supervised interest point detection and description[C]. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Salt Lake City, USA, 2018: 224–236. doi: 10.1109/CVPRW.2018.00060.
[6]	SARLIN P E, DETONE D, MALISIEWICZ T, et al. SuperGlue: Learning feature matching with graph neural networks[C]. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, USA, 2020: 4938–4947. doi: 10.1109/CVPR42600.2020.00499.
[7]	SUN Jiaming, SHEN Zehong, WANG Yuang, et al. LoFTR: Detector-free local feature matching with transformers[C]. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, USA, 2021: 8922–8931. doi: 10.1109/CVPR46437.2021.00881.
[8]	LINDENBERGER P, SARLIN P E, and POLLEFEYS M. LightGlue: Local feature matching at light speed[C]. 2023 IEEE/CVF International Conference on Computer Vision, Paris, France, 2023: 17627–17638. doi: 10.1109/ICCV51070.2023.01616.
[9]	WANG Yifan, HE Xingyi, PENG Sida, et al. Efficient LoFTR: Semi-dense local feature matching with sparse-like speed[C]. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, USA, 2024: 21666–21675. doi: 10.1109/CVPR52733.2024.02047.
[10]	郭志强, 汪子涵, 王永圣, 等. LFTA: 轻量级特征提取与加性注意力的特征匹配方法[J]. 电子与信息学报, 2025, 47(8): 2872–2882. doi: 10.11999/JEIT250124. GUO Zhiqiang, WANG Zihan, WANG Yongsheng, et al. LFTA: Lightweight feature extraction and additive attention-based feature matching method[J]. Journal of Electronics & Information Technology, 2025, 47(8): 2872–2882. doi: 10.11999/JEIT250124.
[11]	徐昌定, 刘世杰, 肖长江. 顾及灰度-梯度双通道特征与形变参数优化的陆标匹配方法[J]. 电子与信息学报, 2025, 47(12): 4754–4762. doi: 10.11999/JEIT250953. XU Changding, LIU Shijie, and XIAO Changjiang. A landmark matching method considering gray-gradient dual-channel features and deformation parameter optimization[J]. Journal of Electronics & Information Technology, 2025, 47(12): 4754–4762. doi: 10.11999/JEIT250953.
[12]	POTJE G, CADAR F, ARAUJO A, et al. XFeat: Accelerated features for lightweight image matching[C]. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, USA, 2024: 2682–2691. doi: 10.1109/CVPR52733.2024.00259.
[13]	SYAWALUDIN M F, KIM S, and HWANG J I. Planar-equirectangular image stitching[J]. Electronics, 2021, 10(9): 1126. doi: 10.3390/electronics10091126.
[14]	JUNG D, CHOI J, LEE Y, et al. EDM: Equirectangular projection-oriented dense kernelized feature matching[C]. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, USA, 2025: 6337–6347. doi: 10.1109/CVPR52734.2025.00594.
[15]	LIANG Anbang, LI Qingquan, CHEN Zhipeng, et al. Spherically optimized RANSAC aided by an IMU for fisheye image matching[J]. Remote Sensing, 2021, 13(10): 2017. doi: 10.3390/rs13102017.
[16]	FISCHLER M A and BOLLES R C. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography[J]. Communications of the ACM, 1981, 24(6): 381–395. doi: 10.1145/358669.358692.
[17]	SOLÀ J, DERAY J, and ATCHUTHAN D. A micro Lie theory for state estimation in robotics[EB/OL]. https://arxiv.org/abs/1812.01537, 2026.
[18]	陈晓雷, 王兴, 张学功, 等. 面向360度全景图像显著目标检测的相邻协调网络[J]. 电子与信息学报, 2024, 46(12): 4529–4541. doi: 10.11999/JEIT240502. CHEN Xiaolei, WANG Xing, ZHANG Xuegong, et al. Adjacent coordination network for salient object detection in 360 degree omnidirectional images[J]. Journal of Electronics & Information Technology, 2024, 46(12): 4529–4541. doi: 10.11999/JEIT240502.
[19]	TU Diantao, CUI Hainan, ZHENG Xianwei, et al. PanoPose: Self-supervised relative pose estimation for panoramic images[C]. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, USA, 2024: 20009–20018. doi: 10.1109/CVPR52733.2024.01891.
[20]	LO I C, SHIH K T, and CHEN H H. Efficient and accurate stitching for 360° dual-fisheye images and videos[J]. IEEE Transactions on Image Processing, 2022, 31: 251–262. doi: 10.1109/TIP.2021.3130531.
[21]	GAVA C, MUKUNDA V, HABTEGEBRIAL T, et al. SphereGlue: Learning keypoint matching on high resolution spherical images[C]. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Vancouver, Canada, 2023: 6134–6144. doi: 10.1109/CVPRW59228.2023.00653.
[22]	KABSCH W. A solution for the best rotation to relate two sets of vectors[J]. Acta Crystallographica Section A, 1976, 32(5): 922–923. doi: 10.1107/S0567739476001873.
[23]	Ricoh Imaging Company, Ltd. RICOH THETA S specifications[EB/OL]. https://www.ricoh-imaging.co.jp/english/products/theta_s/, 2026.