An SO(3)-Manifold-Constrained Registration Method for Twin-Fisheye Panoramic Images

WANG Zhuopeng; LIN Shanling; LIN Jianpu; LÜ Shanhong; LIN Zhixian

doi:10.11999/JEIT260798

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2026 >

WANG Zhuopeng, LIN Shanling, LIN Jianpu, LÜ Shanhong, LIN Zhixian. An SO(3)-Manifold-Constrained Registration Method for Twin-Fisheye Panoramic Images[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260798

Citation:

WANG Zhuopeng, LIN Shanling, LIN Jianpu, LÜ Shanhong, LIN Zhixian. An SO(3)-Manifold-Constrained Registration Method for Twin-Fisheye Panoramic Images[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260798

Citation:

PDF( 3845 KB)

An SO(3)-Manifold-Constrained Registration Method for Twin-Fisheye Panoramic Images

doi: 10.11999/JEIT260798 cstr: 32379.14.JEIT260798

1.
School of Advanced Manufacturing, Fuzhou University, Quanzhou 362251, China
2.
Fujian Science & Technology Innovation Laboratory for Optoelectronic Information of China, Fuzhou 350116, China

Funds: The National Key Research and Development Program of China (Grant No. 2021YFB3600603)

Received Date: 2026-06-16
Accepted Date: 2026-08-13
Rev Recd Date: 2026-08-11

Available Online: 2026-08-15

Abstract

Abstract

Objective Twin-fisheye cameras provide near-360° coverage with low hardware complexity and are widely used in immersive imaging, surveillance, and mobile robotics. Their panoramic output depends on registration over a narrow overlapping band, so geometric accuracy and temporal consistency directly affect seam quality and video smoothness. After the two fisheye views are unfolded into the Equirectangular Projection (ERP), three coupled problems arise. First, the near-co-centric lens pair is ideally related by a pure rotation R ∈SO(3), whereas a conventional 8-Degree-of-Freedom (DoF) homography introduces five redundant parameters that may couple with matching noise. Second, ERP sampling is nonuniform with latitude, so identical pixel residuals do not represent identical spherical angular errors. Third, the cyclic ±π longitude boundary splits structures that are continuous on the sphere and weakens correspondences around the seam. Existing planar pipelines and generic learned matchers rarely combine these constraints under a unified rotation-referenced evaluation. This study therefore develops a lightweight registration framework that explicitly exploits spherical rotation geometry while addressing ERP boundary discontinuity, temporal fluctuation, and long-tail residuals. Methods The proposed framework contains three modules (Fig. 1). First, an overlapping-band Region-of-Interest (ROI) is cropped around the ERP seam and rearranged with modulo-W wrap-around (Fig. 2). A default longitude half-width of ±15° and an approximately 3° margin on each side preserve cross-boundary feature continuity while restricting the search region. XFeat detects, describes, and matches features under a fixed Top-K budget. Second, the two-dimensional matches are restored to global ERP coordinates, mapped to unit-sphere direction vectors, and processed by rotation-only SO(3)-RANSAC. Spherical angular residuals are used as the inlier criterion with a 0.8° threshold, a maximum of 2000 iterations, confidence 0.999, and at least 12 inliers; the iteration bound is updated adaptively. All inliers are then used for Kabsch/SVD closed-form rotation re-estimation, which reduces the randomness of a minimal sample while preserving the SO(3) constraint. Third, a local increment on the Lie algebra so(3) is optimized under a Huber loss by the Levenberg–Marquardt algorithm. The refinement is triggered only when the inlier-residual P95 exceeds 0.90° and the inlier ratio is below 0.58, thereby concentrating nonlinear optimization on difficult image pairs. Results and Discussions Experiments are conducted on PanoraMIS Sequences 3 and 4 under a unified relative inter-frame rotation protocol (Table 1). The proposed method achieves a 97.10% success rate, a 0.549° P95 angular residual, and a 0.313° temporal-stability error. Compared with SuperPoint+LightGlue, the P95 and temporal-stability errors are reduced by 22.8% and 77.3%, respectively. Compared with Efficient LoFTR, peak GPU memory and runtime are reduced by 52.7% and 60.9%, although Efficient LoFTR retains the lowest overall P95. Under a unified SO(3)-RANSAC back-end (Table 2), XFeat provides the largest average inlier count of 625.4 and the lowest temporal-stability error of 0.313° at 38.04 ms. The ablation and sensitivity results (Tables 3–4) show that the ±15° ROI reduces the P95 from 0.720° for the full ERP to 0.600°. Replacing H-RANSAC with SO(3)-RANSAC reduces temporal instability from 0.931° to 0.313°, a 66.3% reduction, while increasing runtime from 24.91 ms to 35.24 ms. Adaptive refinement operates on approximately one third of the image pairs and improves both P95 and temporal stability with lower overhead than always-on refinement; its five-seed mean and median trigger rate are both 36.23%. A Top-K budget of 2048 reaches the saturated accuracy level, because increasing the budget to 4096 yields no further P95 or stability improvement. Five fixed-seed repetitions produce standard deviations no greater than 0.011° for P95 and stability, indicating that the main conclusions are insensitive to RANSAC randomness. In a 5×5 threshold sweep, the maximum changes in P95 and inter-frame rotation jitter within the central neighborhood are 4.40% and 0.037%, respectively, and changing the robust-error truncation from 3° to 2° or 5° does not alter the relative ranking. On the more difficult Sequence 4, characterized by weak texture and unstable overlap, the proposed method obtains 919.7 average inliers and a P95 of 0.383°, the lowest among the evaluated learning-based matchers, although robust-estimation time increases. With an identical standardized stitching back-end, it produces a lower seam-band gradient than H-RANSAC in the small-rotation example (26.90 versus 28.71) and a lower truth-referenced temporal-stability error (0.313° versus 0.931°; Fig. 6). On outdoor Sequence 7-L2, 324 of 346 correspondences are retained, yielding a 93.6% inlier ratio and a 0.524° P95 residual (Fig. 5). Because sequence-specific calibration is unavailable and the fixed inter-lens baseline may cause depth-dependent parallax, this result serves only as a diagnostic consistency check, not as evidence of absolute pose accuracy. Conclusions By restoring feature continuity across the ERP boundary, replacing the redundant planar homography with an explicit SO(3) rotation model, and selectively refining difficult image pairs on so(3), the proposed method balances registration accuracy, temporal consistency, and resource cost. It provides a lightweight front-end for twin-fisheye panorama stitching. The current evaluation is limited to pairwise registration on a small number of sequences; future work will address translation compensation, multi-frame global optimization, end-to-end integration with seam finding, exposure compensation, and blending, as well as generalization across additional platforms, dynamic scenes, and illumination conditions.
- Twin-fisheye images,
- Panoramic stitching,
- Equirectangular projection,
- Lightweight feature matching,
- SO(3)-RANSAC

FullText(HTML)

References(23)

References

[1]	SZELISKI R. Image alignment and stitching: A tutorial[J]. Foundations and Trends^® in Computer Graphics and Vision, 2006, 2(1): 1–104. doi: 10.1561/0600000009.
[2]	BENSEDDIK H E, MORBIDI F, and CARON G. PanoraMIS: An ultra-wide field of view image dataset for vision-based robot-motion estimation[J]. The International Journal of Robotics Research, 2020, 39(9): 1037–1051. doi: 10.1177/0278364920915248.
[3]	LOWE D G. Distinctive image features from scale-invariant keypoints[J]. International Journal of Computer Vision, 2004, 60(2): 91–110. doi: 10.1023/B:VISI.0000029664.99615.94.
[4]	RUBLEE E, RABAUD V, KONOLIGE K, et al. ORB: An efficient alternative to SIFT or SURF[C]. 2011 International Conference on Computer Vision, Barcelona, Spain, 2011: 2564–2571. doi: 10.1109/ICCV.2011.6126544.
[5]	DETONE D, MALISIEWICZ T, and RABINOVICH A. SuperPoint: Self-supervised interest point detection and description[C]. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Salt Lake City, USA, 2018: 224–236. doi: 10.1109/CVPRW.2018.00060.
[6]	SARLIN P E, DETONE D, MALISIEWICZ T, et al. SuperGlue: Learning feature matching with graph neural networks[C]. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, USA, 2020: 4938–4947. doi: 10.1109/CVPR42600.2020.00499.
[7]	SUN Jiaming, SHEN Zehong, WANG Yuang, et al. LoFTR: Detector-free local feature matching with transformers[C]. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, USA, 2021: 8922–8931. doi: 10.1109/CVPR46437.2021.00881.
[8]	LINDENBERGER P, SARLIN P E, and POLLEFEYS M. LightGlue: Local feature matching at light speed[C]. 2023 IEEE/CVF International Conference on Computer Vision, Paris, France, 2023: 17627–17638. doi: 10.1109/ICCV51070.2023.01616.
[9]	WANG Yifan, HE Xingyi, PENG Sida, et al. Efficient LoFTR: Semi-dense local feature matching with sparse-like speed[C]. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, USA, 2024: 21666–21675. doi: 10.1109/CVPR52733.2024.02047.
[10]	郭志强, 汪子涵, 王永圣, 等. LFTA: 轻量级特征提取与加性注意力的特征匹配方法[J]. 电子与信息学报, 2025, 47(8): 2872–2882. doi: 10.11999/JEIT250124. GUO Zhiqiang, WANG Zihan, WANG Yongsheng, et al. LFTA: Lightweight feature extraction and additive attention-based feature matching method[J]. Journal of Electronics & Information Technology, 2025, 47(8): 2872–2882. doi: 10.11999/JEIT250124.
[11]	徐昌定, 刘世杰, 肖长江. 顾及灰度-梯度双通道特征与形变参数优化的陆标匹配方法[J]. 电子与信息学报, 2025, 47(12): 4754–4762. doi: 10.11999/JEIT250953. XU Changding, LIU Shijie, and XIAO Changjiang. A landmark matching method considering gray-gradient dual-channel features and deformation parameter optimization[J]. Journal of Electronics & Information Technology, 2025, 47(12): 4754–4762. doi: 10.11999/JEIT250953.
[12]	POTJE G, CADAR F, ARAUJO A, et al. XFeat: Accelerated features for lightweight image matching[C]. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, USA, 2024: 2682–2691. doi: 10.1109/CVPR52733.2024.00259.
[13]	SYAWALUDIN M F, KIM S, and HWANG J I. Planar-equirectangular image stitching[J]. Electronics, 2021, 10(9): 1126. doi: 10.3390/electronics10091126.
[14]	JUNG D, CHOI J, LEE Y, et al. EDM: Equirectangular projection-oriented dense kernelized feature matching[C]. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, USA, 2025: 6337–6347. doi: 10.1109/CVPR52734.2025.00594.
[15]	LIANG Anbang, LI Qingquan, CHEN Zhipeng, et al. Spherically optimized RANSAC aided by an IMU for fisheye image matching[J]. Remote Sensing, 2021, 13(10): 2017. doi: 10.3390/rs13102017.
[16]	FISCHLER M A and BOLLES R C. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography[J]. Communications of the ACM, 1981, 24(6): 381–395. doi: 10.1145/358669.358692.
[17]	SOLÀ J, DERAY J, and ATCHUTHAN D. A micro Lie theory for state estimation in robotics[EB/OL]. https://arxiv.org/abs/1812.01537, 2026.
[18]	陈晓雷, 王兴, 张学功, 等. 面向360度全景图像显著目标检测的相邻协调网络[J]. 电子与信息学报, 2024, 46(12): 4529–4541. doi: 10.11999/JEIT240502. CHEN Xiaolei, WANG Xing, ZHANG Xuegong, et al. Adjacent coordination network for salient object detection in 360 degree omnidirectional images[J]. Journal of Electronics & Information Technology, 2024, 46(12): 4529–4541. doi: 10.11999/JEIT240502.
[19]	TU Diantao, CUI Hainan, ZHENG Xianwei, et al. PanoPose: Self-supervised relative pose estimation for panoramic images[C]. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, USA, 2024: 20009–20018. doi: 10.1109/CVPR52733.2024.01891.
[20]	LO I C, SHIH K T, and CHEN H H. Efficient and accurate stitching for 360° dual-fisheye images and videos[J]. IEEE Transactions on Image Processing, 2022, 31: 251–262. doi: 10.1109/TIP.2021.3130531.
[21]	GAVA C, MUKUNDA V, HABTEGEBRIAL T, et al. SphereGlue: Learning keypoint matching on high resolution spherical images[C]. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Vancouver, Canada, 2023: 6134–6144. doi: 10.1109/CVPRW59228.2023.00653.
[22]	KABSCH W. A solution for the best rotation to relate two sets of vectors[J]. Acta Crystallographica Section A, 1976, 32(5): 922–923. doi: 10.1107/S0567739476001873.
[23]	Ricoh Imaging Company, Ltd. RICOH THETA S specifications[EB/OL]. https://www.ricoh-imaging.co.jp/english/products/theta_s/, 2026.